Object Storage Swift - Swift commands Memo (EN)

Bazy wiedzy

KB0047244

Object Storage Swift - Swift commands Memo (EN)


Icons/System/eye-open Created with Sketch. 316 wyśw. 16.11.2022 Cloud / Object Storage S3

Objective

You can use the OpenStack API to generate various scripts to automate your actions on your Public Cloud instances.

The OpenStack swiftclient allows you to interact with and manage your containers and objects. For example, you can send files regularly to your containers, in order to back them up.

In this guide, you can find the main swift commands for managing your object containers using python-swiftclient.

Requirements

Instructions

Create a PCS container

swift post <container>

View account information

swift stat

View container information

swift stat <container>

View object information

swift stat <container> <object>
swift list

List the contents of a container

swift list <container>

Upload an object smaller than 5GB

swift upload <container> <object>

Upload an object higher than 5GB in SLO mode

swift upload --use-slo --segment-size 1G <container> <object>

Upload an object higher than 5GB in DLO mode

swift upload --segment-size 1G <container> <object>

LargeObject upload abortion

$ swift upload --use-slo --segment-size 500M <container> <object>
^C Aborted
$ swift list
<container>
<container_segments>
$ swift list <container>
$
$ swift list <container_segments>
<object>/slo/1628738591.297565/6442450944/524288000/00000000
<object>/slo/1628738591.297565/6442450944/524288000/00000001
<object>/slo/1628738591.297565/6442450944/524288000/00000002
<object>/slo/1628738591.297565/6442450944/524288000/00000003
<object>/slo/1628738591.297565/6442450944/524288000/00000004
<object>/slo/1628738591.297565/6442450944/524288000/00000005
<object>/slo/1628738591.297565/6442450944/524288000/00000006
<object>/slo/1628738591.297565/6442450944/524288000/00000007
<object>/slo/1628738591.297565/6442450944/524288000/00000008
<object>/slo/1628738591.297565/6442450944/524288000/00000009

It is recommended that you delete the <container_segments> or at least the segments that correspond to the dropped object.

Download an object

swift download <container> <object>

Delete an empty container

swift delete <container>

Delete a non-empty container

swift delete <container>

Delete an object

swift delete <container> <object>

Delete prefix objects

swift delete --prefix <prefix> <container>

Add a metadata to a container

swift post -H "X-Container-Meta-Access-Control-Allow-Origin:http://example.com" <container>

Add metadata to an object

swift post -m "my-custom-key:value" <container> <object>

Delete a metadata from a container

swift post -H "X-Container-Meta-Access-Control-Allow-Origin:" <container>

Delete a metadata from an object

swift post -m "my-custom-key:" <container> <object>
swift post -H "X-Remove-Object-My-Custom-Key:" <container> <object>

Set Read ACL on a container

swift post <container> -r "${OS_TENANT_ID}:*"

Set write ACL on a container

swift post <container> -w "${OS_TENANT_ID}:*"

Delete the read ACL on a container

swift post <container> -r ""

Delete write ACL on a container

swift post <container> -w ""

Move objects from one container to another

swift copy -d /<destination_container> <container> <object>

LargeObjects

In this example, the LargeObject was uploaded in SLO mode. See the Curl command memo documentation for the manifest upload.

On a LargeObject, the swift copy command returns a 413 error:

swift copy -d /<destination_container> <container> <largeobject>
created container <cdestination_container>
Object COPY failed: https://storage.gra.cloud.ovh.net/v1/AUTH_702xxxxxxxxxxxxxxxxxxxxxxxxxxdaf/<container>/<largeobject> 413 Request Entity Too Large  [first 60 chars of response] b'<html><h1>Request Entity Too Large</h1><p>The body of your r'

So we need to start by moving the segments:

for obj in $(swift list <container_segments>);do swift copy -d /<destination_container_segments> <container_segments> $obj;done
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000000 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000000
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000001 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000001
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000002 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000002
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000003 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000003
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000004 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000004
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000005 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000005

Then retrieve the manifest, adapt it and re-upload it

$(swift auth)
curl -s -X GET "$OS_STORAGE_URL/<container>/<largeobject>?multipart-manifest=get" -H "X-Auth-Token:$OS_AUTH_TOKEN" | jq '.' > <largeobject>.json

sed -i 's/name/path/g' <largeobject>.json
sed -i 's/bytes/size_bytes/g' <largeobject>.json
sed -i '/hash/d' <largeobject>.json
sed -i '/last_modified/d' <largeobject>.json
sed -i '/content_type/d' <largeobject>.json
sed -i '/path/s/,$//g' <largeobject>.json

curl -i -X PUT -H "X-Auth-Token:$OS_AUTH_TOKEN" -T <largeobject>.json "$OS_STORAGE_URL/<destination_container>/<largeobject>?multipart-manifest=put`

Rename a container

You cannot rename a container. To do this, create a new container and re-upload the objects in it. However, Swift has the copy feature, which seems to provide better performance.

Time to upload 2Gb (1500 objects of 1M and 1 object of 500M):

time swift upload <container> ./*
real    69m26,159s
user    0m20,017s
sys 0m3,689s
swift list --lh -t <container>
2.0G
swift list
<container>
time for obj in $(swift list <container>);do swift copy -d /<other_container> <container> $obj; done
real    54m43,898s
user    12m38,060s
sys 1m34,394s
swift list
<other_container>
<container>
swift list --lh -t <other_container>
2.0G

We notice that for small objects, this does not change much. However, for objects of a slightly larger size (500M):

time swift upload <container> <object> --object-name <bis_object>
<bis_object>

real    15m51,525s
user    0m4,245s
sys 0m0,848s
time swift copy -d /<other_container> <container> <bis_object>
created container <other_container>
<container>/<bis_object> copied to /<other_container>/<bis_object>

real    0m11,924s
user    0m0,464s
sys 0m0,091s

LargeObject

In this example, the LargeObject was uploaded in SLO mode. Please refer to the Curl Command Memo documentation for the manifest upload. 

time swift upload --use-slo --segment-size 1G <container> <largeobject>
<largeobject> segment 4
<largeobject> segment 2
<largeobject> segment 0
<largeobject> segment 1
<largeobject> segment 5
<largeobject> segment 3
<largeobject>

real    190m55,547s
user    0m57,906s
sys 0m14,246s

On a LargeObject, the swift copy command returns a 413 error:

swift copy -d /<destination_container> <container> <largeobject>
created container <cdestination_container>
Object COPY failed: https://storage.gra.cloud.ovh.net/v1/AUTH_702xxxxxxxxxxxxxxxxxxxxxxxxxxdaf/<container>/<largeobject> 413 Request Entity Too Large [first 60 chars of response] b'<html><h1>Request Entity Too Large</h1><p>The body of your r'

So we need to start by moving the segments:

for obj in $(swift list <container_segments>);do swift copy -d /<destination_container_segments> <container_segments> $obj;done
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000000 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000000
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000001 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000001
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000002 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000002
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000003 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000003
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000004 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000004
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000005 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000005

Then retrieve the manifest, adapt it and re-upload it

$(swift auth)
curl -s -X GET "$OS_STORAGE_URL/<container>/<largeobject>?multipart-manifest=get" -H "X-Auth-Token:$OS_AUTH_TOKEN" | jq '.' ><largeobject>.json

sed -i 's/name/path/g' <largeobject>.json
sed -i 's/bytes/size_bytes/g' <largeobject>.json
sed -i '/hash/d' <largeobject>.json
sed -i '/last_modified/d' <largeobject>.json
sed -i '/content_type/d' <largeobject>.json
sed -i '/path/s/,$//g' <largeobject>.json

curl -i -X PUT -H "X-Auth-Token:$OS_AUTH_TOKEN" -T <largeobject>.json "$OS_STORAGE_URL/<destination_container>/<largeobject>?multipart-manifest=put"

Get the space used in a container

swift list --lh -t <container>
8.4G

There is no concept of a folder in a container, however we can use the prefixes:

swift list -p <prefix> <container>
<prefix>/1.jpg
<prefix>/10.jpg
<prefix>/11.jpg
<prefix>/12.jpg
<prefix>/13.jpg
<prefix>/14.jpg
<prefix>/15.jpg
<prefix>/16.jpg
<prefix>/17.jpg
<prefix>/18.jpg
<prefix>/2.jpg
<prefix>/3.jpg
<prefix>/4.jpg
<prefix>/5.jpg
<prefix>/6.jpg
<prefix>/7.jpg
<prefix>/8.jpg
<prefix>/9.jpg
swift list --lh -t -p <prefix> <container>
685K

Get user id

openstack user show --format json "${OS_USERNAME}" | jq -r '.id'
openstack token issue -f json | jq -r '.user_id'

Get cluster limits

swift capabilities

Limit container size

To limit the size of a container, simply add the metadata 'X-Container-Meta-Quota-Bytes' associated with the limit to set up:

swift post -H "X-Container-Meta-Quota-Bytes:<limit-in-bytes>" <container>

If the limit is exceeded, the following error is returned:

swift upload <container> logo_ovh.png
Object PUT failed: https://storage.gra.cloud.ovh.net/v1/AUTH_<account>/<container>/logo_ovh.png 413 Request Entity Too Large b'Upload exceeds quota.' (txn: txee1f2b77c26a424ebbda8-0062a1aa7f)
Consider using the --segment-size option to chunk the object

Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for assisting you on your specific use case of your project.

Join our community of users.

Powiązane artykuły