Cloudmesh

• 1: Introduction
• 2: Installation
• 3: Output
• 4: Dictionaries
• 5: Shell
• 6: StopWatch
• 7: Cloudmesh Command Shell
• 8: Exercises

1 - Introduction

Gregor von Laszewski (laszewski@gmail.com)

Learning Objectives

• Introduction to the cloudmesh API
• Using cmd5 via cms
• Introduction to cloudmesh convenience API for output, dotdict, shell, stopwatch, benchmark management
• Creating your own cms commands
• Cloudmesh configuration file
• Cloudmesh inventory

In this chapter, we like to introduce you to cloudmesh which provides you with a number of convenient methods to interface with the local system, but also with cloud services. We will start while focussing on some simple APIs and then gradually introduce the cloudmesh shell which not only provides a shell but also a command line interface so you can use cloudmesh from a terminal. This dual ability is quite useful as we can write cloudmesh scripts, but can also invoke the functionality from the terminal. This is quite an important distinction from other tools that only allow command line interfaces.

Moreover, we also show you that it is easy to create new commands and add them dynamically to the cloudmesh shell via simple pip installs.

Cloudmesh is an evolving project and you have the opportunity to improve if you see some features missing.

The manual of cloudmesh can be found at

• https://cloudmesh.github.io/cloudmesh-manual

The API documentation is located at

• https://cloudmesh.github.io/cloudmesh-manual/api/index.html#cloudmesh-api

We will initially focus on a subset of this functionality.

2 - Installation

Gregor von Laszewski (laszewski@gmail.com)

The installation of cloudmesh is simple and can technically be done via pip by a user. However you are not a user, you are a developer. Cloudmesh is distributed in different topical repositories and in order for developers to easily interact with them we have written a convenient cloudmesh-installer program.

As a developer, you must also use a python virtual environment to avoid affecting your system-wide Python installation. This can be achieved while using Python3 from python.org or via conda. However, we do recommend that you use python.org as this is the vanilla python that most developers in the world use. Conda is often used by users of Python if they do not need to use bleeding-edge but older prepackaged Python tools and libraries.

Prerequisite

We require you to create a python virtual environment and activate it. How to do this was discussed in sec. ¿sec:python-install?. Please create the ENV3 environment. Please activate it.

Basic Install

Cloudmesh can install for developers a number of bundles. A bundle is a set of git repositories that are needed for a particular install. For us, we are mostly interested in the bundles cms, cloud, storage. We will introduce you to other bundles throughout this documentation.

If you like to find out more about the details of this you can look at cloudmesh-installer which will be regularly updated.

To make use of the bundle and the easy installation for developers please install the cloudmesh-installer via pip, but make sure you do this in a python virtual env as discussed previously. If not you may impact your system negatively. Please note that we are not responsible for fixing your computer. Naturally, you can also use a virtual machine, if you prefer. It is also important that we create a uniform development environment. In our case, we create an empty directory called cm in which we place the bundle.

$ mkdir cm
$ cd cm
$ pip install cloudmesh-installer

To see the bundle you can use

$ cloudmesh-installer bundles

We will start with the basic cloudmesh functionality at this time and only install the shell and some common APIs.

$ cloudmesh-installer git clone cms
$ cloudmesh-installer install cms

These commands download and install cloudmesh shell into your environment. It is important that you use the -e flag

To see if it works you can use the command

$ cms help

You will see an output. If this does not work for you, and you can not figure out the issue, please contact us so we can identify what went wrong.

For more information, please visit our Installation Instructions for Developers

3 - Output

Gregor von Laszewski (laszewski@gmail.com)

Cloudmesh provides a number of convenient API’s to make the output easier or more fanciful.

These API’s include

• Console
• Banner
• Heading
• VERBOSE

Console

Print is the usual function to output to the terminal. However, often we like to have colored output that helps us in the notification to the user. For this reason, we have a simple Console class that has several built-in features. You can even switch and define your own color schemes.

from cloudmesh.common.console import Console

msg = "my message"
Console.ok(msg) # prins a green message
Console.error(msg) # prins a red message proceeded with ERROR
Console.msg(msg) # prins a regular black message

In case of the error message we also have convenient flags that allow us to include the traceback in the output.

Console.error(msg, prefix=True, traceflag=True)

The prefix can be switched on and off with the prefix flag, while the traceflag switches on and of if the trace should be set.

The verbosity of the output is controlled via variables that are stored in the ~/.cloudmesh directory.

from cloudmesh.common.variables import Variables

variables = Variables()

variables['debug'] = True
variables['trace'] = True
variables['verbose'] = 10

For more features, see API: Console

Banner

In case you need a banner you can do this with

from cloudmesh.common.util import banner

banner("my text")

For more features, see API: Banner

Heading

A particularly useful function is HEADING() which prints the method name.

from cloudmesh.common.util import HEADING

class Example(object):

    def doit(self):
        HEADING()
        print ("Hello")

The invocation of the HEADING() function doit prints a banner with the name information. The reason we did not do it as a decorator is that you can place the HEADING() function in an arbitrary location of the method body.

For more features, see API: Heading

VERBOSE

Note: VERBOSE is not supported in jupyter notebooks

VERBOSE is a very useful method allowing you to print a dictionary. Not only will it print the dict, but it will also provide you with the information in which file it is used and which line number. It will even print the name of the dict that you use in your code.

To use this you will have to enable the debugging methods for cloudmesh as discussed in sec. 1.1

from cloudmesh.common.debug import VERBOSE

m = {"key": "value"}
VERBOSE(m)

For more features, please see VERBOSE

Using print and pprint

In many cases, it may be sufficient to use print and pprint for debugging. However, as the code is big and you may forget where you placed print statements or the print statements may have been added by others, we recommend that you use the VERBOSE function. If you use print or pprint we recommend using a unique prefix, such as:

from pprint import pprint

d = {"sample": "value"}
print("MYDEBUG:")
pprint (d)

# or with print

print("MYDEBUG:", d)

4 - Dictionaries

Gregor von Laszewski (laszewski@gmail.com)

Dotdict

For simple dictionaries we sometimes like to simplify the notation with a . instead of using the []:

You can achieve this with dotdict

from cloudmesh.common.dotdict import dotdict

data = {
    "name": "Gregor"
}

data = dotdict(data)

Now you can either call

data["name"]

or

data.name

This is especially useful in if conditions as it may be easier to read and write

if data.name is "Gregor":

    print("this is quite readable")

and is the same as

if data["name"] is "Gregor":

    print("this is quite readable")

For more features, see API: dotdict

FlatDict

In some cases, it is useful to be able to flatten out dictionaries that contain dicts within dicts. For this, we can use FlatDict.

from cloudmesh.common.Flatdict import FlatDict

data = {
    "name": "Gregor",
    "address": {
        "city": "Bloomington",
        "state": "IN"

    }
}

flat = FlatDict(data, sep=".")

This will be converted to a dict with the following structure.

flat = {
    "name": "Gregor"
    "address.city": "Bloomington",
    "address.state": "IN"
}

With sep you can change the sepaerator between the nested dict attributes. For more features, see API: dotdict

Printing Dicts

In case we want to print dicts and lists of dicts in various formats, we have included a simple Printer that can print a dict in yaml, json, table, and csv format.

The function can even guess from the passed parameters what the input format is and uses the appropriate internal function.

A common example is

from pprint import pprint
from cloudmesh.common.Printer import Printer

data = [
    {
        "name": "Gregor",
        "address": {
            "street": "Funny Lane 11",
            "city": "Cloudville"
        }
    },
    {
        "name": "Albert",
        "address": {
            "street": "Memory Lane 1901",
            "city": "Cloudnine"
        }
    }
]


pprint(data)

table = Printer.flatwrite(data,
                          sort_keys=["name"],
                          order=["name", "address.street", "address.city"],
                          header=["Name", "Street", "City"],
                          output='table')

print(table)

For more features, see API: Printer

More examples are available in the source code as tests

5 - Shell

Gregor von Laszewski (laszewski@gmail.com)

Python provides a sophisticated method for starting background processes. However, in many cases, it is quite complex to interact with it. It also does not provide convenient wrappers that we can use to start them in a pythonic fashion. For this reason, we have written a primitive Shell class that provides just enough functionality to be useful in many cases.

Let us review some examples where result is set to the output of the command being executed.

from cloudmesh.common.Shell import Shell

result = Shell.execute('pwd')
print(result)

result = Shell.execute('ls', ["-l", "-a"])
print(result)

result = Shell.execute('ls', "-l -a")
print(result)

For many common commands, we provide built-in functions. For example:

result = Shell.ls("-aux")
print(result)

result = Shell.ls("-a", "-u", "-x")
print(result)

result = Shell.pwd()
print(result)

The list includes (naturally the commands that must be available on your OS. If the shell command is not available on your OS, please help us improving the code to either provide functions that work on your OS or develop with us platform-independent functionality of a subset of the functionality for the shell command that we may benefit from.

• VBoxManage(cls, *args)
• bash(cls, *args)
• blockdiag(cls, *args)
• brew(cls, *args)
• cat(cls, *args)
• check_output(cls, *args, **kwargs)
• check_python(cls)
• cm(cls, *args)
• cms(cls, *args)
• command_exists(cls, name)
• dialog(cls, *args)
• edit(filename)
• execute(cls,*args)
• fgrep(cls, *args)
• find_cygwin_executables(cls)
• find_lines_with(cls, lines, what)
• get_python(cls)
• git(cls, *args)
• grep(cls, *args)
• head(cls, *args)
• install(cls, name)
• install(cls, name)
• keystone(cls, *args)
• kill(cls, *args)
• live(cls, command, cwd=None)
• ls(cls, *args)
• mkdir(cls, directory)
• mongod(cls, *args)
• nosetests(cls, *args)
• nova(cls, *args)
• operating_system(cls)
• pandoc(cls, *args)
• ping(cls, host=None, count=1)
• pip(cls, *args)
• ps(cls, *args)
• pwd(cls, *args)
• rackdiag(cls, *args)
• remove_line_with(cls, lines, what)
• rm(cls, *args)
• rsync(cls, *args)
• scp(cls, *args)
• sh(cls, *args)
• sort(cls, *args)
• ssh(cls, *args)
• sudo(cls, *args)
• tail(cls, *args)
• terminal(cls, command='pwd')
• terminal_type(cls)
• unzip(cls, source_filename, dest_dir)
• vagrant(cls, *args)
• version(cls, name)
• which(cls, command)

For more features, please see Shell

6 - StopWatch

Gregor von Laszewski (laszewski@gmail.com)

Often you find yourself in a situation where you like to measure the time between two events. We provide a simple StopWatch that allows you not only to measure a number of times but also to print them out in a convenient format.

from cloudmesh.common.StopWatch import StopWatch
from time import sleep


StopWatch.start("test")
sleep(1)
StopWatch.stop("test")

print (StopWatch.get("test"))

To print, you can simply also use:

StopWatch.benchmark()

For more features, please see StopWatch

7 - Cloudmesh Command Shell

Gregor von Laszewski (laszewski@gmail.com)

CMD5

Python’s CMD (https://docs.python.org/2/library/cmd.html) is a very useful package to create command line shells. However, it does not allow the dynamic integration of newly defined commands. Furthermore, additions to CMD need to be done within the same source tree. To simplify developing commands by a number of people and to have a dynamic plugin mechanism, we developed cmd5. It is a rewrite of our earlier efforts in cloudmesh client and cmd3.

Resources

The source code for cmd5 is located in GitHub:

• https://github.com/cloudmesh/cmd5

We have discussed in sec. ¿sec:cloudmesh-cms-install? how to install cloudmesh as a developer and have access to the source code in a directory called cm. As you read this document we assume you are a developer and can skip the next section.

Installation from source

WARNING: DO NOT EXECUTE THIS IF YOU ARE A DEVELOPER OR YOUR ENVIRONMENT WILL NOT PROPERLY WORK. YOU LIKELY HAVE ALREADY INSTALLED CMD5 IF YOU USED THE CLOUDMESH INSTALLER.

However, if you are a user of cloudmesh you can install it with

$ pip install cloudmesh-cmd5

Execution

To run the shell you can activate it with the cms command. cms stands for cloudmesh shell:

(ENV2) $ cms

It will print the banner and enter the shell:

+-------------------------------------------------------+
|   ____ _                 _                     _      |
|  / ___| | ___  _   _  __| |_ __ ___   ___  ___| |__   |
| | |   | |/ _ \| | | |/ _` | '_ ` _ \ / _ \/ __| '_ \  |
| | |___| | (_) | |_| | (_| | | | | | |  __/\__ \ | | | |
|  \____|_|\___/ \__,_|\__,_|_| |_| |_|\___||___/_| |_| |
+-------------------------------------------------------+
|                  Cloudmesh CMD5 Shell                 |
+-------------------------------------------------------+

cms>

To see the list of commands you can say:

cms> help

To see the manual page for a specific command, please use:

help COMMANDNAME

Create your own Extension

One of the most important features of CMD5 is its ability to extend it with new commands. This is done via packaged namespaces. We recommend you name it cloudmesh-mycommand, where mycommand is the name of the command that you like to create. This can easily be done while using the sys* cloudmesh command (we suggest you use a different name than gregor maybe your firstname):

$ cms sys command generate gregor

It will download a template from cloudmesh called cloudmesh-bar and generate a new directory cloudmesh-gregor with all the needed files to create your own command and register it dynamically with cloudmesh. All you have to do is to cd into the directory and install the code:

$ cd cloudmesh-gregor
$ python setup.py install
# pip install .

Adding your command is easy. It is important that all objects are defined in the command itself and that no global variables be used to allow each shell command to stand alone. Naturally, you should develop API libraries outside of the cloudmesh shell command and reuse them to keep the command code as small as possible. We place the command in:

cloudmsesh/mycommand/command/gregor.py

Now you can go ahead and modify your command in that directory. It will look similar to (if you used the command name gregor):

from cloudmesh.shell.command import command
from cloudmesh.shell.command import PluginCommand

class GregorCommand(PluginCommand):

    @command
    def do_gregor(self, args, arguments):
        """
        ::
          Usage:
                gregor -f FILE
                gregor list
          This command does some useful things.
          Arguments:
              FILE   a file name
          Options:
              -f      specify the file
        """
        print(arguments)
        if arguments.FILE:
           print("You have used file: ", arguments.FILE)
        return ""

An important difference to other CMD solutions is that our commands can leverage (besides the standard definition), docopts as a way to define the manual page. This allows us to use arguments as dict and use simple if conditions to interpret the command. Using docopts has the advantage that contributors are forced to think about the command and its options and document them from the start. Previously we did not use but argparse and click. However, we noticed that for our contributors both systems lead to commands that were either not properly documented or the developers delivered ambiguous commands that resulted in confusion and wrong usage by subsequent users. Hence, we do recommend that you use docopts for documenting cmd5 commands. The transformation is enabled by the @command decorator that generates a manual page and creates a proper help message for the shell automatically. Thus there is no need to introduce a separate help method as would normally be needed in CMD while reducing the effort it takes to contribute new commands in a dynamic fashion.

Bug: Quotes

We have one bug in cmd5 that relates to the use of quotes on the commandline

For example, you need to say

$ cms gregor -f \"file name with spaces\"

If you like to help us fix this that would be great. it requires the use of shlex. Unfortunately, we did not yet time to fix this “feature.”

8 - Exercises

Gregor von Laszewski (laszewski@gmail.com)

When doing your assignment, make sure you label the programs appropriately with comments that clearly identify the assignment. Place all assignments in a folder on GitHub named “cloudmesh-exercises”

For example, name the program solving E.Cloudmesh.Common.1 e-cloudmesh-1.py and so on. For more complex assignments you can name them as you like, as long as in the file you have a comment such as

# fa19-516-000 E.Cloudmesh.Common.1

at the beginning of the file. Please do not store any screenshots in your GitHub repository of your working program.

Cloudmesh Common

E.Cloudmesh.Common.1

Develop a program that demonstrates the use of banner, HEADING, and VERBOSE.

E.Cloudmesh.Common.2

Develop a program that demonstrates the use of dotdict.

E.Cloudmesh.Common.3

Develop a program that demonstrates the use of FlatDict.

E.Cloudmesh.Common.4

Develop a program that demonstrates the use of cloudmesh.common.Shell.

E.Cloudmesh.Common.5

Develop a program that demonstrates the use of cloudmesh.common.StopWatch.

Cloudmesh Shell

E.Cloudmesh.Shell.1

Install cmd5 and the command cms on your computer.

E. Cloudmesh.Shell.2

Write a new command with your firstname as the command name.

E.Cloudmesh.Shell.3

Write a new command and experiment with docopt syntax and argument interpretation of the dict with if conditions.

E.Cloudmesh.Shell.4

If you have useful extensions that you like us to add by default, please work with us.

E.Cloudmesh.Shell.5

At this time one needs to quote in some commands the " in the shell command line. Develop and test code that fixes this.