Using the s3cmd utility

Introduction

After we have installed s3cmd and set up the configuration file, we can start working with our repository.
In this article, we will analyze the basic commands and examples of using the s3cmd utility.

Basic Commands

Creating a Bucket

To create a bucket, we use the mb (make bucket) command.

NB! In our example, the bucket name is “test” and we use the previously created configuration file “example_config.cfg“

s3cmd mb s3://test -c example_config.cfg

Uploading a file to a bucket

Use the put command to upload files to a bucket.

For example, let’s create a text file with the name testfile.txt and the content “test“

echo 'test' > testfile.txt

Uploading a file to a bucket

s3cmd put testfile.txt s3://test -c example_config.cfg

If the download was successful, we will see the following message:

upload: 'testfile.txt' -> 's3://test/testfile.txt'  [1 of 1]
 5 of 5   100% in    0s    96.12 B/s  done

To upload multiple files at once, use the following syntax:

s3cmd put testfile.txt testfile2.txt testfile3.txt s3://test -c example_config.cfg

Downloading files from a bucket

To copy files from a bucket, use the get command.

s3cmd get s3://test/testfile.txt -c example_config.cfg

If you need to copy several files at once, add the –recursive flag, and add a trailing slash to the end of the s3 address.

s3cmd get s3://test/ --recursive -c example_config.cfg

If you want to copy a file and save it locally under a different name, write the new name immediately after the file you are copying from the repository.

s3cmd get s3://test/testfile.txt newtestfile.txt -c example_config.cfg

Permissions

Let’s try to access the “testfile” file previously uploaded to the bucket through the browser or with the curl command:

curl -i https://test.s3.pilw.io/testfile.txt
HTTP/1.1 403 Forbidden

Getting a file access denied

Set the access to the file as public

s3cmd setacl s3://test/testfile.txt --acl-public -c example_config.cfg

We get the following message:

s3://test/testfile.txt: ACL set to Public [1 of 1]

Let’s try to access the file again through the browser or with the curl command

curl -i https://test.s3.pilw.io/testfile.txt
HTTP/1.1 200 OK
...

test

The file is now read successfully

To make a file private again, use the following command:

s3cmd setacl s3://example/testfile.txt --acl-private -c example_config.cfg

We get the following message:

s3://test/testfile.txt: ACL set to Private [1 of 1]

Let’s try to access the file again through the browser or with the curl command

curl -i https://test.s3.pilw.io/testfile.txt
HTTP/1.1 403 Forbidden

We get access denied, the file has become private again.

Delete

To remove a file from a bucket, use the rm or del command:

s3cmd rm s3://test/testfile.txt -c example_config.cfg
s3cmd del s3://test/testfile.txt -c example_config.cfg

To remove files from the repository, issue the rm or del command, using the –recursive and –force flags

s3cmd rm s3://test  --recursive --force -c example_config.cfg

To remove an empty bucket, use the rb (remove bucket) command. If you need to delete the repository with all its contents, add the –recursive flag

s3cmd rb s3://test -c example_config.cfg
s3cmd rb s3://test --recursive  -c example_config.cfg

View content

To view a list of all created repositories, use the ls command:

s3cmd ls -c example_config.cfg

If you need to recursively view the contents of one or more buckets, you must specify these buckets in the command:

s3cmd ls s3://test s3://test1 -c example_config.cfg

To display the contents of all existing repositories, use the command:

s3cmd ls --recursive -c example_config.cfg

Data Sync

The sync command writes data buffered in memory to disk.

Let’s create some files and folders with files, and upload one of the folders to our bucket.
We upload the folder1 folder to our storage, to upload the folder with all the contents, we use the recursive flag.

s3cmd put --recursive folder1 s3://test/ -c example_config.cfg
upload: 'folder1/folderfile1.txt' -> 's3://test/folder1/folderfile1.txt' [1 of 1]
5 of 5 100% in 0s 94.85 B/s done

The folder was successfully uploaded to the bucket. You can check this with the ls command:

s3cmd ls s3://test/ -c example_config.cfg

Now let’s use the sync command:

s3cmd sync ./ s3://test/ -c example_config.cfg
upload: './folder2/folderfile2.txt' -> 's3://test/folder2/folderfile2.txt' [1 of 3]
5 of 5 100% in 0s 95.67 B/s done
upload: './testfile1.txt' -> 's3://test/testfile1.txt' [3 of 3]
15 of 15 100% in 0s 38.32 B/s done
upload: './testfile2.txt' -> 's3://test/testfile1.txt' [3 of 3]
15 of 15 100% in 0s 280.76 B/s done
Done. Uploaded 2147 bytes in 1.0 seconds, 2.10 KB/s.

As you can see, only the files that were not previously uploaded to the bucket were synchronized.
S3cmd checks file sizes and checksums.
If changes occur, the file will be overwritten when resynchronizing.

Browser Access

There are two ways to access the file in the browser:

https://test.s3.pilw.io/testfile.txt
https://s3.pilw.io/test/testfile.txt

test — bucket name
testfile.txt – the name of the file we are accessing

Conclusion

In the article, you got acquainted with some basic commands of the s3cmd utility.

s3cmd offers many more commands. More information about the usage guide and other utility commands can be found in the official documentation.