Programmable Storage Without The Programming

As data centers scale, the ability to automate routine tasks enables this growth to occur without requiring additional staff. The problem is most automation requires learning a programming or scripting language which most IT professionals don’t have the time to do. As a result, they either keep manually doing tasks over and over one at a time, or hire an outside consultant to automate the tasks for them.

Vendors, of course, are sensitive to the problem and are continually adding features or capabilities to their solutions. But these additions are added based on popularity and practicality. The process doesn’t take into account the uniqueness of every data center. Nothing provides customization like a programming or scripting language nor does anything require the learning curve!

Nexenta may have an answer. Both its NexentaStor and NexentaEdge management APIs have a REST front end that enables the automation of many routine tasks as well as provides insights into the operation of the storage systems. To get around the programming curve, they provide the browser-based Swagger interface. With it, IT administrators can configure and retrieve information about their NexentaStor and NexentaEdge environments. There are a couple of sample scripts in the comments section of this post.

Nexenta Management API Use Cases

For cloud and managed service providers, the Nexenta Management APIs gives them the ability to automate such tasks as client on-boarding, capacity expansion and migration to alternate systems or clusters. They can integrate this automation with their own portal software, providing a seamless experience to customers.

Enterprises can use the management API for Test/Dev situations to automate the provisioning of new copies or snapshots of data so that developers are always working with the latest copy of code. Like service providers, they can automate the provisioning of new VMs or even help with capacity and resource planning. The automation also allows enterprises to increase performance of tasks like backup jobs by running simple pre and post scripts to ensure they get full use of their hardware.

The Advantages of Swagger

Swagger not only makes it easy for the non-programmer to quickly interact with their Nexenta systems, it also provides a self-documenting, central repository of automation for them to leverage across the enterprise. And potentially more important, unlike legacy scripting approaches, it does not require access to the kernel layer, making it more secure.

StorageSwiss Take

Creating a cloud is more than just a heavily virtualized infrastructure or an infrastructure that uses a modern software defined or scale out storage solution. True clouds are also highly automated, provide self-service and orchestrated provisioning.

While some vendors have tried to bake these capabilities into their standard user interface, eventually the data center requires very specific customization. The only way to get to that point is through scriptable automation of which REST seems to be the most popular. The challenge is how to learn, document and share those projects. By working with Swagger, Nexenta enables its customers to quickly jump on the automation fast track to lower IT costs while improving service levels.

Sponsored by Nexenta

Eight years ago George Crump, founded Storage Switzerland with one simple goal. To educate IT professionals about all aspects of data center storage. He is the primary contributor to Storage Switzerland and is and a heavily sought after public speaker. With 25 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS and SAN, Virtualization, Cloud and Enterprise Flash. Prior to founding Storage Switzerland he was CTO at one the nation's largest storage integrators where he was in charge of technology testing, integration and product selection.

Tagged with: , , , , , ,
Posted in Blog
2 comments on “Programmable Storage Without The Programming
  1. George Crump says:

    Here is a samle script for creating and sharing a filesystem

    #!/usr/bin/perl -w
    use strict;
    use JSON::Tiny qw(decode_json encode_json);;
    use URL::Encode qw(url_encode_utf8);;

    ##
    ## Demo code using curl to call NS5 API
    ## Author: Ryuji Masuda
    ##
    ## API Documented on NS5 https://$serverip:8443/docs/
    ## To use API enable swagger from CLI – >config set rest.Swagger=true
    ##

    ## Setup login info and IP Address of NS5 storagenode
    ##

    my $username = “admin”;
    my $password = “nexentA01”;
    my $server_addr = “https://172.16.251.132:8443”;
    my $curlheader = q[curl -s -k -H “Accept: application/json” -H “Content-Type: application/json”];

    ## Folder we want to operate on.
    my $folder = “data/testfs”;

    ## Functions for actions such as auth, creating/destroying filesystem and shares
    ##
    sub setup_auth {

    my ($username,$password,$server_addr,$curlheader) = @_;

    my $bytes = encode_json {username => $username, password => $password};

    my $cmd = qq[$curlheader -X POST -d ‘$bytes’ $server_addr/auth/login];
    my $string = `$cmd`;
    my $hash = decode_json($string);
    my $token = $$hash{token};
    $curlheader .= qq[ -H “Authorization: Bearer $token” ];

    return $curlheader;

    }

    sub create_fs {
    my($pathname) = @_;
    my $bytes = encode_json { path => $pathname };
    my $cmd = qq[$curlheader -X POST -d ‘$bytes’ $server_addr/storage/filesystems];
    `$cmd`;

    }

    sub create_nfsshare {
    my($pathname) = @_;
    my $bytes = encode_json { filesystem => $pathname };
    my $cmd = qq[$curlheader -X POST -d ‘$bytes’ $server_addr/nas/nfs];
    `$cmd`;

    }

    sub delete_nfsshare {
    my($pathname) = @_;
    $pathname = url_encode_utf8($pathname);
    my $cmd = qq[$curlheader -X DELETE $server_addr/nas/nfs/$pathname];
    `$cmd`;

    }

    sub delete_fs {
    my($pathname) = @_;
    $pathname = url_encode_utf8($pathname);
    my $cmd = qq[$curlheader -X DELETE $server_addr/storage/filesystems/$pathname];
    `$cmd`;

    }

    ##
    ##
    ##
    ## Authenticate against the API and setup the token to pass to Curl
    ##

    $curlheader = setup_auth($username,$password,$server_addr,$curlheader);

    ## Create a folder called /data/testfs and export it via NFS
    ##
    create_fs($folder);
    create_nfsshare($folder);

    ## Unshare the filesystem then destroy
    ##

    delete_nfsshare($folder);
    delete_fs($folder);

    ##my $poweroff = qq[$curlheader -X POST $server_addr/node/powerOff];
    ##`$poweroff`;

  2. George Crump says:

    Here is a class for interacting with the NEF REST API.

    Copyright (C) 2016 Nexenta Systems
    William Kettler
    “””

    import logging
    import requests
    import json

    logger = logging.getLogger(__name__)

    class NEFClient(object):
    “””
    NEF REST API client.

    WARNING this class does not currently validate the SSL certificate.

    Attributes:
    url (str): API url, i.e. https://
    port (int): API port
    username (str): Optional username, required if password provided
    password (str): Optional password, required if username provided
    “””

    def __init__(self):
    self.url = “http://localhost:8080”
    self.username = None
    self.password = None
    self.key = None
    self.verify = False
    self.headers = {
    “Content-Type”: “application/json”
    }

    # Disable security warnings
    #requests.packages.urllib3.disable_warnings()

    # Lets auth now so we can exit immediately if credentials are bad
    if self.username is not None and self.password is not None:
    self._login()
    elif self.username is not None:
    raise TypeError(“A password is required when username is provided”)
    elif self.password is not None:
    raise TypeError(“A username is required when password is provided”)

    def _login(self):
    “””
    Sends a login request.
    “””
    method = “auth/login”
    payload = {
    “username”: self.username,
    “password”: self.password
    }

    logger.debug(“Logging in as user %s to %s”, self.username, self.url)
    try:
    response = requests.post(“/”.join([self.url, method]),
    data=payload, verify=self.verify)
    response.raise_for_status()
    body = response.json()
    # Bookmark until I find out what error handling makes sense
    except:
    raise

    logger.debug(body)

    self.key = body[“token”]
    self.headers[“Authorization”] = “Bearer %s” % self.key

    def logout(self):
    “””
    Sends logout request.
    “””
    method = “auth/logout”
    logger.debug(“Logging out as user %s on %s”, self.username, self.url)
    self.post(method)

    def get(self, method, params=None):
    “””
    Sends a GET request.

    Args:
    method (str): NEF API method
    Kwargs:
    params (dict): Request parameters
    Returns:
    The HTML response body as a dict.
    “””
    logger.debug(“GET %s”, method)
    logger.debug(params)
    try:
    response = requests.get(“/”.join([self.url, method]),
    headers=self.headers, verify=self.verify,
    params=params)
    response.raise_for_status()
    # Bookmark until I find out what error handling makes sense
    except:
    raise

    # If there is no response body json() will fail
    try:
    body = response.json()
    except ValueError:
    body = None

    logger.debug(body)

    return body

    def post(self, method, payload=None):
    “””
    Sends a POST request.

    Args:
    method (str): NEF API method
    Kwargs:
    payload (dict): Request payload
    Returns:
    The job ID if the request is ASYNC otherwise None.
    “””
    logger.debug(“POST %s”, method)
    logger.debug(payload)
    try:
    response = requests.post(“/”.join([self.url, method]),
    headers=self.headers, verify=self.verify,
    data=json.dumps(payload))
    response.raise_for_status()
    # Bookmark until I find out what error handling makes sense
    except:
    raise

    # If there is no response body json() will fail
    try:
    body = response.json()
    except ValueError:
    body = None

    # If the status code is 202 the request is in progress
    if response.status_code == 202:
    jobid = body[“links”][0][“href”].split(“/”)[-1]
    else:
    jobid = None

    logger.debug(body)

    return jobid

    def put(self, method, payload=None):
    “””
    Sends a PUT request.

    Args:
    method (str): NEF API method
    Kwargs:
    payload (dict): Request payload
    Returns:
    The job ID if the request is ASYNC otherwise None.
    “””
    logger.debug(“PUT %s”, method)
    logger.debug(payload)
    try:
    response = requests.put(“/”.join([self.url, method]),
    headers=self.headers, verify=self.verify,
    data=json.dumps(payload))
    response.raise_for_status()
    # Bookmark until I find out what error handling makes sense
    except:
    raise

    # If there is no response body json() will fail
    try:
    body = response.json()
    except ValueError:
    body = None

    # If the status code is 202 the request is in progress
    if response.status_code == 202:
    jobid = body[“links”][0][“href”].split(“/”)[-1]
    else:
    jobid = None

    logger.debug(body)

    return jobid

    def delete(self, method, payload=None):
    “””
    Sends a DELETE request.

    Args:
    method (str): NEF API method
    Kwargs:
    payload (dict): Request payload
    Returns:
    The job ID if the request is ASYNC otherwise None.
    “””
    logger.debug(“DELETE %s”, method)
    logger.debug(payload)
    try:
    response = requests.delete(“/”.join([self.url, method]),
    headers=self.headers,
    verify=self.verify,
    data=json.dumps(payload))
    response.raise_for_status()
    # Bookmark until I find out what error handling makes sense
    except:
    raise

    # If there is no response body json() will fail
    try:
    body = response.json()
    except ValueError:
    body = None

    # If the status code is 202 the request is in progress
    if response.status_code == 202:
    jobid = body[“links”][0][“href”].split(“/”)[-1]
    else:
    jobid = None

    logger.debug(body)

    return jobid

    def jobstatus(self, jobid):
    “””
    Determine the ASYNC job status.

    Args:
    jobid (str): The job ID returned by the ASYN request
    Returns:
    The job ID progress and state (i.e. done or not).
    “””
    method = “jobStatus”
    params = {
    “jobId”: jobid
    }

    body = self.get(method, params=params)
    try:
    progress = body[“data”][0][“progress”]
    done = body[“data”][0][“done”]
    except IndexError:
    raise RuntimeError(“The job ID no longer exists”)

    return done, progress

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 21,604 other followers

Blog Stats
  • 924,223 views
%d bloggers like this: