Accessing IBM Object Store from Python

IBM Object Store

SWIFT Object StoreIBM offers a S3 compatible Object Store as a file storage. Beside S3 the storage can also be accessed via the SWIFT protocol by selecting a different deploy model. As the cost for this storage is extremely low compared to Database storage it is perfect for storing sensor data or other kind of data for machine learning.

I use the storage for example to host my training data or trained model for Tensorflow. Access and payment for the Object Store is managed via IBM Cloud aka Bluemix. And as this offering is included in the Lite offering the first 25GB are for free. 🙂

As there is a problem getting the S3 credentials right now I use the SWIFT access model. Please make sure when you request the Object Store service to access the SWIFT version to select the right access model.

Python libs

As the SWIFT protocol is part of openstack, the python access client can be found at https://docs.openstack.org/python-swiftclient. Depending on the security access model you also need the openstack Identity API (Keystone). Both libs are on github (swiftclient and keystone) and also available via pip.

Access storage

Inside the IBM Cloud web interface you can create or read existing credentials. If your program runs on IBM Cloud (Cloudfoundry or Kubernetes) the credentials are also available via the VCAP environment variable. In both cases they look like mine here:

Important informations are the projectId, region, userId and password. The access with keystone the swift python client looks like this:

Important is the version information, also as part of the authurl.

Accessing data

Objects can be read and written, containers (aka buckets) can we read and modified as described in the documentation. For example:

 

 

Docker container with Bluemix CLI tools

BluemixCLI on docker hub
BluemixCLI on docker hub

Being an developer advocate means to play always with the latest version of tools and being on the edge. But installed programs are getting out of date and so I always end up with having installed old versions of CLI tools. One reason why I love cloud (aka other people’s computers) computing so much is because I don’t need to update the software, it is done by professionals. In order to have always the latest version of my Bluemix CLI tools in hand and being authenticated I compiled a little docker container with my favourite command line tools. cf, bx, docker and wsk.

Getting the docker container

I published the docker container on the official docker hub. So getting it is very easy when the docker tools are installed. This command will download the latest version of the container and therefore the latest version of installed cli tools. We need to run this command from time to time to make sure the latest version is available on our computer.

Get the necessary parameters

For all command line tools we need username, passwords and IDs. Obviously we can not hardcode them into the docker container therefore we need to pass them along as command line parameters when starting the docker container.

  • Username (the same as we use to login to Bluemix)
  • Password (the same as we use to login to Bluemix)
  • Org (The Organisation we want to work in, must already be existing)
  • Space (The Space we want to work in, must already be created)
  • AccountID (This can we catched from the URL when we open “Manage Organisation and click on the account)
  • OpenwhiskID (Individual for org and space, get be catched here: https://console.bluemix.net/openwhisk/learn/cli)

Run the container

The container can be started with docker run and passing all parameters with -e in:

Line 8 mounts the local directory inside the docker container under /root/host. So we can fire up the container and have a bash with the latest tools and our source code available.

Use the tools

Before we can use the tools we need to configure them and authenticate against Bluemix. The script “init.sh” which is located in “/root/” (our working directory) takes care of all logins and authentications.

cf

The Cloudfoundry command line tool for starting, stopping apps and connecting services.

bx

The Bluemix version of the Cloudfoundry command line tool. Including the plugin for container maintenance. By initializing this plugin we also get the credentials and settings for the docker client to use Bluemix as a docker daemon.

docker

The normal docker client with Bluemix as daemon configured.

wsk

The OpenWhisk client already authenticated.

We can configure an alias in our .bashrc so by just typing “bxdev” we will have bash with the latest cli tools available.