Etherarp
Etherarp

Networking, Security, Linux

Share


Using S3 for temporary storage

I often use S3 for ad-hoc storage. This post will look at configuring auto-deletion of objects, as well as generating temporary public URLs

Rohan MolloyRohan Molloy

Configure a bucket to auto-delete objects

In this example, a bucket will be configured to store objects for no longer than a week, deleting them thereafter.

$ s3cmd mb s3://ephemeral-bucket
Bucket 's3://ephemeral-bucket/' created

$ s3cmd expire --expiry-days 7 s3://ephemeral-bucket
Bucket 's3://ephemeral-bucket/': expiration configuration is set.

s3cmd expire is quite dumb and will overwrite existing rules each time it's run. For more sophisticated policies, it's better to use aws s3api and pass the lifecycle policy as JSON. In the below example, daily and weekly lifecycle polices are set for the /daily_folder/ and /weekly_folder/ locations, respectively.

$ cat > /tmp/lifecycle.json << EOF
{
 "Rules": [
 {
  "ID": "$(uuidgen)",
  "Filter": {
    "Prefix": "daily_folder/" 
 }, 
 "Status": "Enabled", 
 "Expiration": {
   "Days": 1
  }
 },
 {
  "ID": "$(uuidgen)",
   "Filter": {
     "Prefix": "weekly_folder/"
 }, 
 "Status": "Enabled", 
 "Expiration": {
   "Days": 7
  }
 }
 ]
}
EOF

$ aws s3api
put-bucket-lifecycle-configuration \
--bucket lifecycle-bucket-example \
--lifecycle-configuration file:///tmp/lifecycle.json

Get a temporary public URL for an object

The s3cmd signurl command generates a URL that provides public access to an existing object. The URL is time-limited and non-guessable.

$ expires=$(date -d 'now + 5 day' +%s)
$ date -d @$expires
Mon Apr 1 19:42:52 NZDT 2019

$ s3cmd put example.html s3://ephemeral-bucket/example.html
upload: '/tmp/example.html' -> 's3://ephemeral-bucket/example.html' [1 of 1]
 12264 of 12264 100% in 0s 39.27 kB/s done

$ s3cmd signurl s3://ephemeral-bucket/example.html "$expires"
http://ephemeral-bucket.s3.amazonaws.com/example.html?AWSAccessKeyId=AKIAJTLN47OITGH32KBA&Expires=1554101979&Signature=M0tX10W1RfuahUIXe6qgf9cUl4k%3D




Author

Rohan Molloy

View Comments