IBM Object Store
IBM offers a S3 compatible Object Store as a file storage. Beside S3 the storage can also be accessed via the SWIFT protocol by selecting a different deploy model. As the cost for this storage is extremely low compared to Database storage it is perfect for storing sensor data or other kind of data for machine learning.
I use the storage for example to host my training data or trained model for Tensorflow. Access and payment for the Object Store is managed via IBM Cloud aka Bluemix. And as this offering is included in the Lite offering the first 25GB are for free. 🙂
As there is a problem getting the S3 credentials right now I use the SWIFT access model. Please make sure when you request the Object Store service to access the SWIFT version to select the right access model.
Python libs
As the SWIFT protocol is part of openstack, the python access client can be found at https://docs.openstack.org/python-swiftclient. Depending on the security access model you also need the openstack Identity API (Keystone). Both libs are on github (swiftclient and keystone) and also available via pip.
|
1 2 |
pip install python-swiftclient pip install python-keystoneclient |
Access storage
Inside the IBM Cloud web interface you can create or read existing credentials. If your program runs on IBM Cloud (Cloudfoundry or Kubernetes) the credentials are also available via the VCAP environment variable. In both cases they look like mine here:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
{ "auth_url": "https://identity.open.softlayer.com", "project": "object_storage_xxxxxxxx_xxxx_xxxx_b35a_6d007e3f9118", "projectId": "512xxxxxxxxxxxxxxxxxxxxxe00fe4e1", "region": "dallas", "userId": "e8c19efxxxxxxxxxxxxxxxxxxx91d53e", "username": "admin_1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa66", "password": "fTxxxxxxxxxxw8}l", "domainId": "15xxxxxxxxxxxxxxxxxxxxxxxxxxxx2a", "domainName": "77xxx3", "role": "admin" } |
Important informations are the projectId, region, userId and password. The access with keystone the swift python client looks like this:
|
1 2 3 4 5 6 7 |
conn = Connection(key=VCAP['password'], authurl='https://identity.open.softlayer.com/v3', auth_version='3', os_options={"project_id":=VCAP['projectId'], "user_id":=VCAP['userId'], "region_name": VCAP['region']} ) |
Important is the version information, also as part of the authurl.
Accessing data
Objects can be read and written, containers (aka buckets) can we read and modified as described in the documentation. For example:
|
1 2 3 4 5 6 7 8 9 |
resp_headers, containers = conn.get_account() # Get container conn.put_container('containerName') # Create new container conn.put_object(container, # Write files 'local_object.txt', contents=local, content_type='text/plain' ) resp_headers, obj_contents = conn.get_object('container', 'file.txt') ' conn.delete_object('container', 'file.txt') |
