Using S3 for temporary storage

Configure a bucket to auto-delete objects

In this example, a bucket will be configured to store objects for no longer than a week, deleting them thereafter.

$ s3cmd mb s3://ephemeral-bucket
Bucket 's3://ephemeral-bucket/' created

$ s3cmd expire --expiry-days 7 s3://ephemeral-bucket
Bucket 's3://ephemeral-bucket/': expiration configuration is set.

s3cmd expire is quite dumb and will overwrite existing rules each time it's run. For more sophisticated policies, it's better to use aws s3api and pass the lifecycle policy as JSON. In the below example, daily and weekly lifecycle polices are set for the /daily_folder/ and /weekly_folder/ locations, respectively.

$ cat > /tmp/lifecycle.json << EOF
{
"Rules": [
{
  "ID": "$(uuidgen)",
  "Filter": {
    "Prefix": "daily_folder/"
},
"Status": "Enabled",
"Expiration": {
   "Days": 1
  }
},
{
  "ID": "$(uuidgen)",
   "Filter": {
     "Prefix": "weekly_folder/"
},
"Status": "Enabled",
"Expiration": {
   "Days": 7
  }
}
]
}
EOF

$ aws s3api
put-bucket-lifecycle-configuration \
--bucket lifecycle-bucket-example \
--lifecycle-configuration file:///tmp/lifecycle.json

Get a temporary public URL for an object

The s3cmd signurl command generates a URL that provides public access to an existing object. The URL is time-limited and non-guessable.

$ expires=$(date -d 'now + 5 day' +%s)
$ date -d @$expires
Mon Apr 1 19:42:52 NZDT 2019

$ s3cmd put example.html s3://ephemeral-bucket/example.html
upload: '/tmp/example.html' -> 's3://ephemeral-bucket/example.html' [1 of 1]
12264 of 12264 100% in 0s 39.27 kB/s done

$ s3cmd signurl s3://ephemeral-bucket/example.html "$expires"
http://ephemeral-bucket.s3.amazonaws.com/example.html?AWSAccessKeyId=AKIAJTLN47OITGH32KBA&Expires=1554101979&Signature=M0tX10W1RfuahUIXe6qgf9cUl4k%3D