{"id":381,"date":"2020-02-04T13:13:00","date_gmt":"2020-02-04T13:13:00","guid":{"rendered":"https:\/\/bootstrap-it.com\/blog\/?p=381"},"modified":"2020-08-04T13:29:18","modified_gmt":"2020-08-04T13:29:18","slug":"how-to-use-a-bash-script-to-manage-downloading-and-viewing-files-from-an-aws-s3-bucket","status":"publish","type":"post","link":"https:\/\/bootstrap-it.com\/blog\/?p=381","title":{"rendered":"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket"},"content":{"rendered":"
<\/a><\/a><\/a><\/a><\/a><\/div>\n

As you can read in this article<\/a>, I recently had some trouble with my email server and decided to outsource email administration to Amazon’s Simple Email Service (SES).<\/p>\n\n\n\n

The problem with that solution was that I had SES save new messages to an S3 bucket, and using the AWS Management Console to read files within S3 buckets gets stale really fast.<\/p>\n\n\n\n

So I decided to write a Bash script to automate the process of downloading, properly storing, and viewing new messages.<\/p>\n\n\n\n

While I wrote this script for use on my Ubuntu Linux desktop, it wouldn’t require too much fiddling to make it work on a macOS or Windows 10 system through Windows SubSystem for Linux<\/a>.<\/p>\n\n\n\n

Here’s the complete script all in one piece. After you take a few moments to look it over, I’ll walk you through it one step at a time.<\/p>\n\n\n\n

#!\/bin\/bash\n# Retrieve new messages from S3 and save to tmpemails\/ directory:\naws s3 cp \\\n   --recursive \\\n   s3:\/\/bucket-name\/ \\\n   \/home\/david\/s3-emails\/tmpemails\/  \\\n   --profile myaccount\n\n# Set location variables:\ntmp_file_location=\/home\/david\/s3-emails\/tmpemails\/*\nbase_location=\/home\/david\/s3-emails\/emails\/\n\n# Create new directory to store today's messages:\ntoday=$(date +\"%m_%d_%Y\")\n[[ -d ${base_location}\/\"$today\" ]] || mkdir ${base_location}\/\"$today\"\n\n# Give the message files readable names:\nfor FILE in $tmp_file_location\ndo\n   mv $FILE ${base_location}\/${today}\/email$(rand)\ndone\n\n# Open new files in Gedit:\nfor NEWFILE in ${base_location}\/${today}\/*\ndo\n   gedit $NEWFILE\ndone<\/code><\/pre>\n\n\n\n
The complete Bash script<\/p>\n\n\n\n
We’ll begin with the single command to download any messages currently residing in my S3 bucket (by the way, I’ve changed the names of the bucket and other filesystem and authentication details to protect my privacy).<\/p>\n\n\n\n
aws s3 cp \\\n   --recursive \\\n   s3:\/\/bucket-name\/ \\\n   \/home\/david\/s3-emails\/tmpemails\/  \\\n   --profile myaccount<\/code><\/pre>\n\n\n\n
Of course, this will only work if you’ve already installed and configured the AWS CLI for your local system. Now’s the time to do that<\/a> if you haven’t already.<\/p>\n\n\n\n
The cp<\/em> command stands for “copy,” –recursive<\/em> tells the CLI to apply the operation even to multiple objects, s3:\/\/bucket-name<\/em> points to my bucket (your bucket name will obviously be different), the \/home\/david… line is the absolute filesystem address to which I’d like the messages copied, and the –profile<\/em> argument tells the CLI which of my multiple AWS accounts I’m referring to.<\/p>\n\n\n\n
The next section sets two variables that will make it much easier for me to specify filesystem locations through the rest of the script.<\/p>\n\n\n\n
tmp_file_location=\/home\/david\/s3-emails\/tmpemails\/*\nbase_location=\/home\/david\/s3-emails\/emails\/<\/code><\/pre>\n\n\n\n
Note how the value of the tmp_file_location<\/em> variable ends with an asterisk. That’s because I want to refer to the files within<\/em> that directory, rather than the directory itself.<\/p>\n\n\n\n
I’ll create a new permanent directory within the …\/emails\/ hierarchy to make it easier for me to find messages later. The name of this new directory will be the current date.<\/p>\n\n\n\n
today=$(date +\"%m_%d_%Y\")\n[[ -d ${base_location}\/\"$today\" ]] || mkdir ${base_location}\/\"$today\"<\/code><\/pre>\n\n\n\n
I first create a new shell variable named today<\/em> that will be populated by the output of the date +”%m_%d_%Y”<\/em> command. date<\/em> itself outputs the full date\/timestamp, but what follows (“%m_%d_%Y”<\/em>) edits that output to a simpler and more readable format.<\/p>\n\n\n\n
I then test for the existence of a directly using that name – which would indicate that I’ve already received emails on that day and, therefore, there’s no need to recreate the directory. If such a directory does not<\/em> exist (||), then mkdir<\/em> will create it for me. If you don’t run this test, your command could return annoying error messages.<\/p>\n\n\n\n
Since Amazon SES gives ugly and unreadable names to each of the messages it drops into my S3 bucket, I’ll now dynamically rename them while, at the same time, moving them over to their new home (in the dated directory I just created).<\/p>\n\n\n\n
for FILE in $tmp_file_location\ndo\n   mv $FILE ${base_location}\/${today}\/email$(rand)\ndone<\/code><\/pre>\n\n\n\n
The for…do…done<\/em> loop will read each of the files in the directory represented by the $tmp_file_location<\/em> variable and then move it to the directory I just created (represented by the $base_location<\/em> variable in addition to the current value of $today<\/em>).<\/p>\n\n\n\n
As part of the same operation, I’ll give it its new name, the string “email<\/em>” followed by a random number generated by the rand<\/em> command. You may need to install a random number generator: that’ll be apt install rand <\/em>on Ubuntu.<\/p>\n\n\n\n
An earlier version of the script created names differentiated by shorter, sequential numbers that were incremented using a count=1…count=$((count+1))<\/em> logic within the for<\/em> loop. That worked fine as long as I didn’t happen to receive more than one batch of messages on the same day. If I did, then the new messages would overwrite older files in that day’s directory.<\/p>\n\n\n\n
I guess it’s mathematically possible that my rand<\/em> command could assign overlapping numbers to two files but, given that the default range rand <\/em>uses is between 1 and 32,576, that’s a risk I’m willing to take.<\/p>\n\n\n\n
At this point, there should be files in the new directory with names like email3039, email25343, etc. for each of the new messages I was sent.<\/p>\n\n\n\n
Running the tree<\/em> command on my own system shows me that five messages were saved to my 02_27_2020 directory, and one more to 02_28_2020 (these files were generated using the older version of my script, so they’re numbered sequentially).<\/p>\n\n\n\n
There are currently no files in tmpemails – <\/em>that’s because the mv command moves files to their new location, leaving nothing behind.<\/p>\n\n\n\n
$ tree\n.\n\u251c\u2500\u2500 emails\n\u2502   \u251c\u2500\u2500 02_27_2020\n\u2502   \u2502   \u251c\u2500\u2500 email1\n\u2502   \u2502   \u251c\u2500\u2500 email2\n\u2502   \u2502   \u251c\u2500\u2500 email3\n\u2502   \u2502   \u251c\u2500\u2500 email4\n\u2502   \u2502   \u251c\u2500\u2500 email5\n\u2502   \u2514\u2500\u2500 02_28_2020\n\u2502       \u2514\u2500\u2500 email1\n\u2514\u2500\u2500 tmpemails<\/code><\/pre>\n\n\n\n
The final section of the script opens each new message in my favorite desktop text editor (Gedit). It uses a similar for…do…done<\/em> loop, this time reading the names of each file in the new directory (referenced using the “today<\/em>” command) and then opening the file in Gedit. Note the asterisk I added to the end of the directory location.<\/p>\n\n\n\n
for NEWFILE in ${base_location}\/${today}\/*\ndo\n   gedit $NEWFILE\ndone<\/code><\/pre>\n\n\n\n
There’s still one more thing to do. If I don’t clean out my S3 bucket, it’ll download all the accumulated messages each time I run the script. That’ll make it progressively harder to manage.<\/p>\n\n\n\n
So, after successfully downloading my new messages, I run this short script to delete all the files in the bucket:<\/p>\n\n\n\n
#!\/bin\/bash\n# Delete all existing emails \n\naws s3 rm --recursive s3:\/\/bucket-name\/ --profile myaccount<\/code><\/pre>\n\n\n\n
\n","protected":false},"excerpt":{"rendered":"
As you can read in this article, I recently had some trouble with my email server and decided to outsource email administration to Amazon’s Simple Email Service (SES). The problem with that solution was that I had SES save new messages… Continue Reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":382,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-381","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"yoast_head":"\nHow to use a Bash script to manage downloading and viewing files from an AWS S3 bucket - Bootstrap IT<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bootstrap-it.com\/blog\/?p=381\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket - Bootstrap IT\" \/>\n<meta property=\"og:description\" content=\"As you can read in this article, I recently had some trouble with my email server and decided to outsource email administration to Amazon’s Simple Email Service (SES). The problem with that solution was that I had SES save new messages… Continue Reading →\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bootstrap-it.com\/blog\/?p=381\" \/>\n<meta property=\"og:site_name\" content=\"Bootstrap IT\" \/>\n<meta property=\"article:published_time\" content=\"2020-02-04T13:13:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-08-04T13:29:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/bootstrap-it.com\/blog\/wp-content\/uploads\/code-coding-computer-data-574077.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"199\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"dbclin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@davidbclinton\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"dbclin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/?p=381\",\"url\":\"https:\/\/bootstrap-it.com\/blog\/?p=381\",\"name\":\"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket - Bootstrap IT\",\"isPartOf\":{\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#website\"},\"datePublished\":\"2020-02-04T13:13:00+00:00\",\"dateModified\":\"2020-08-04T13:29:18+00:00\",\"author\":{\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/ae0fb1d5b3b01558b92b6426d77766ec\"},\"breadcrumb\":{\"@id\":\"https:\/\/bootstrap-it.com\/blog\/?p=381#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/bootstrap-it.com\/blog\/?p=381\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/?p=381#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/bootstrap-it.com\/blog\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#website\",\"url\":\"https:\/\/bootstrap-it.com\/blog\/\",\"name\":\"Bootstrap IT\",\"description\":\"Learn technology using technology\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/bootstrap-it.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/ae0fb1d5b3b01558b92b6426d77766ec\",\"name\":\"dbclin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a93785d437350478a7f1dfcbec58d26bc28e0124e405179acbe1b4325c09f90a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a93785d437350478a7f1dfcbec58d26bc28e0124e405179acbe1b4325c09f90a?s=96&d=mm&r=g\",\"caption\":\"dbclin\"},\"sameAs\":[\"http:\/\/bootstrap-it.com\/\",\"dbclinton\",\"https:\/\/twitter.com\/davidbclinton\"],\"url\":\"https:\/\/bootstrap-it.com\/blog\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket - Bootstrap IT","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bootstrap-it.com\/blog\/?p=381","og_locale":"en_US","og_type":"article","og_title":"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket - Bootstrap IT","og_description":"As you can read in this article, I recently had some trouble with my email server and decided to outsource email administration to Amazon’s Simple Email Service (SES). The problem with that solution was that I had SES save new messages… Continue Reading →","og_url":"https:\/\/bootstrap-it.com\/blog\/?p=381","og_site_name":"Bootstrap IT","article_published_time":"2020-02-04T13:13:00+00:00","article_modified_time":"2020-08-04T13:29:18+00:00","og_image":[{"width":300,"height":199,"url":"https:\/\/bootstrap-it.com\/blog\/wp-content\/uploads\/code-coding-computer-data-574077.jpg","type":"image\/jpeg"}],"author":"dbclin","twitter_card":"summary_large_image","twitter_creator":"@davidbclinton","twitter_misc":{"Written by":"dbclin","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/bootstrap-it.com\/blog\/?p=381","url":"https:\/\/bootstrap-it.com\/blog\/?p=381","name":"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket - Bootstrap IT","isPartOf":{"@id":"https:\/\/bootstrap-it.com\/blog\/#website"},"datePublished":"2020-02-04T13:13:00+00:00","dateModified":"2020-08-04T13:29:18+00:00","author":{"@id":"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/ae0fb1d5b3b01558b92b6426d77766ec"},"breadcrumb":{"@id":"https:\/\/bootstrap-it.com\/blog\/?p=381#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bootstrap-it.com\/blog\/?p=381"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/bootstrap-it.com\/blog\/?p=381#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bootstrap-it.com\/blog"},{"@type":"ListItem","position":2,"name":"How to use a Bash script to manage downloading and viewing files from an AWS S3 bucket"}]},{"@type":"WebSite","@id":"https:\/\/bootstrap-it.com\/blog\/#website","url":"https:\/\/bootstrap-it.com\/blog\/","name":"Bootstrap IT","description":"Learn technology using technology","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bootstrap-it.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/ae0fb1d5b3b01558b92b6426d77766ec","name":"dbclin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a93785d437350478a7f1dfcbec58d26bc28e0124e405179acbe1b4325c09f90a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a93785d437350478a7f1dfcbec58d26bc28e0124e405179acbe1b4325c09f90a?s=96&d=mm&r=g","caption":"dbclin"},"sameAs":["http:\/\/bootstrap-it.com\/","dbclinton","https:\/\/twitter.com\/davidbclinton"],"url":"https:\/\/bootstrap-it.com\/blog\/?author=1"}]}},"_links":{"self":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/381","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=381"}],"version-history":[{"count":1,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/381\/revisions"}],"predecessor-version":[{"id":383,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/381\/revisions\/383"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/media\/382"}],"wp:attachment":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=381"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=381"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=381"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}