DZone Snippets: Amazon code

Rake task to set all S3 files public_read

spiceee (Spiceee) — Tue, 15 Jul 2008 01:00:37 GMT

// If you ever need to make sure all your Amazon S3 files are set to public_read, here's a rake task





namespace :s3 do

  desc "Make all objects in S3 public_read"

  task :make_public_readable do

    require 'aws/s3'

    # you might have this setup as env vars, doesn't work for me as i have more than one AWS account

    AWS::S3::Base.establish_connection!(:access_key_id => '',:secret_access_key => '')    

    

    marker = ""

    

    loop do

      objects = AWS::S3::Bucket.objects('your_bucket', :marker=>marker, :max_keys=>100)

      puts "found #{objects.size} objects"

    

      break if objects.size == 0

    

      marker = objects.last.key

      puts "new marker is \"#{marker}\""

    

      public_grant = AWS::S3::ACL::Grant.grant :public_read

  

      objects.each do |o|

        if not o.acl.grants.include? public_grant

          puts "\"#{o.key}\" does not include public_read"

          o.acl.grants << public_grant

          o.acl(o.acl)

        end

      end

    end

  end

  

end

Creating a bucket in Amazon S3 through an irb session

jrobertson (James Robertson) — Tue, 29 Apr 2008 17:20:25 GMT

1) Log into an irb session, and enter your S3 login details.



require 'rubygems'

require 'aws/s3'



  AWS::S3::Base.establish_connection!(

    :access_key_id     => 'REPLACE_ME',

    :secret_access_key => 'REPLACE_ME'

  )

output:
=> #, @secret_access_key="", @options={:server=>"s3.amazonaws.com", :access_key_id=>"", :port=>80, :secret_access_key=>"", :persistent=>true}, @access_key_id="19S45GYAGWK8DC2B8VG2">

2) Browse the existing buckets.
AWS::S3::Service.buckets
output:
=> [#"ogg.twitteraudio.com", "creation_date"=>Sat Apr 26 10:40:16 UTC 2008}>, #"t1000", "creation_date"=>Fri Apr 25 21:35:21 UTC 2008}>, #"t2000", "creation_date"=>Fri Apr 25 21:53:15 UTC 2008}>]

3) Browse the buckets in a programmatical way.
AWS::S3::Service.buckets.each {|b| puts b.name}
output:
ogg.twitteraudio.com
t1000
t2000

4) Add a new bucket called t3000.
AWS::S3::Bucket.create('t3000')
output:
=> true

5) Observe adding the bucket again doesn't cause an error.
AWS::S3::Bucket.create('t3000')
output:
=> true

6) View the buckets again.
AWS::S3::Service.buckets
output:
=> [#"ogg.twitteraudio.com", "creation_date"=>Sat Apr 26 10:40:16 UTC 2008}>, #"t1000", "creation_date"=>Fri Apr 25 21:35:21 UTC 2008}>, #"t2000", "creation_date"=>Fri Apr 25 21:53:15 UTC 2008}>]

Note: You would expect t3000 to be in there however it didn't appear possibly because of the bucket permissions.

7) Let's then look for bucket t3000.
t3000 = AWS::S3::Bucket.find('t3000')
output:
=> #nil, "name"=>"t3000", "marker"=>nil, "max_keys"=>1000, "is_truncated"=>false, "xmlns"=>"http://s3.amazonaws.com/doc/2006-03-01/"}>

8) Now that we've found the bucket let's upload a text file called works.txt.
file = "works.txt"
output:
=> "works.txt"
AWS::S3::S3Object.store(file, open(file), 't3000', :access => :public_read)
output:
=> #

9) Setting the file access to :public_read allows us to view the file from the http location http://t3000.s3.amazonaws.com/works.txt

References:
http://amazon.rubyforge.org/
upload_to_s3 - Ruby S3 upload client [dzone.com]

*update: 14:30 30 April 2008 *
I didn't use Bucket.objects(:reload) which is the reason why the bucket t3000 didn't show up with the statement Service.buckets

Reference: spatten design - Amazon S3, Ruby and Rails slides [spattendesign.com]

Using ruby to upload multiple files to Amazon S3

jrobertson (James Robertson) — Sat, 26 Apr 2008 14:17:58 GMT

This ruby code builds upon the code S3 upload client for Ruby [dzone.com] by reading the account details and file details from XML files.



#!/usr/bin/env ruby

#file : rubys3file2upload.rb



require 'rubygems'

require 'aws/s3'

require 'rexml/document'

include REXML



class S3FileUpload



  def initialize(xml_accounts_file, xml_upload_file)

    initialize_account(xml_accounts_file)

    main(xml_upload_file)

  end



  def initialize_account(accounts_file)



    file = File.new(accounts_file)

    doc = Document.new(file)



    account = doc.root.elements['records/access']

    h = Hash.new

    h[:access_key_id] = account.elements['key_id'].text.to_s

    h[:secret_access_key] = account.elements['secret'].text.to_s

    AWS::S3::Base.establish_connection!(h)



  end



  def main(xml_upload_file)

    file = File.new(xml_upload_file)

    doc = Document.new(file)

puts doc

    doc.root.elements.each('records/file') do |f|

      local_file = f.elements['local'].text.to_s

      bucket = f.elements['bucket'].text.to_s

      mime_type = f.elements['mime_type'].text.to_s

      upload(local_file, mime_type, bucket)

    end

  end



  def upload(local_file, mime_type, bucket)



    base_name = File.basename(local_file)



    puts "Uploading #{local_file} as '#{base_name}' to '#{bucket}'"



    AWS::S3::S3Object.store(

      base_name,

      File.open(local_file),

      bucket,

      :content_type => mime_type,

      :access => :public_read

    )



    puts "Uploaded!"

  end



end



if __FILE__ == $0

  s3fu = S3FileUpload.new('s3accounts.xml', 's3files2upload.xml')

end

file: s3accounts.xml



  

    REPLACE_MEREPLACE_MEREPLACE_ME

file: s3files2upload.xml



  

    

      autocomplete.html

      t2000

      text/html

    

    

      autocomplete2.html

      t2000

      text/html

    

    

      autocomplete3.html

      t2000

      text/html

Note: The file given the correct permission could be read from http://t2000.s3.amazonaws.com/autocomplete.html

Reference: http://amazon.rubyforge.org/

Simple S3 utils - copy bucket to bucket

wiseleyb (ben) — Sun, 30 Dec 2007 20:13:56 GMT

Some simple S3 utils in Ruby

Specifically written to copy one bucket to another (for testing on production on staging)

PLEASE BE CAREFUL WITH THIS - THERE IS CODE THAT DELETES ALL CONTENTS OF A BUCKET

Example:



a = AmazoneS3Asset.new

a.copy_over_bucket("myapp_production", "myapp_production")



require 'aws/s3'

require 'mechanize'



class AmazonS3Asset

  

  include AWS::S3

  S3ID = "your s3 id"

  S3KEY = "your s3 key"

  

  def initialize

    puts "connecting..."

    AWS::S3::Base.establish_connection!(

      :access_key_id     => S3ID,

      :secret_access_key => S3KEY

    )

  end



  def delete_key(bucket, key)

    if exists?(bucket, key) 

      S3Object.delete key, bucket

    end

  end

  

  def empty_bucket(bucket)

    bucket_keys(bucket).each do |k|

      puts "deleting #{k}"

      delete_key(bucket,k)

    end

  end

  

  def bucket_keys(bucket)

    b = Bucket.find(bucket)

    b.objects.collect {|o| o.key}

  end



  def copy_over_bucket(from_bucket, to_bucket)

    puts "Replacing #{to_bucket} with contents of #{from_bucket}"

    #delete to_bucket

    empty_bucket(to_bucket)

    bucket_keys(from_bucket).each do |k|

      copy_between_buckets(from_bucket, to_bucket, k)

    end

  end

  

  def copy_between_buckets(from_bucket, to_bucket, from_key, to_key = nil)

    if exists?(from_bucket, from_key)

      to_key = from_key if to_key.nil?

      puts "Copying #{from_bucket}.#{from_key} to #{to_bucket}.#{to_key}"

      url = "http://s3.amazonaws.com/#{from_bucket}/#{from_key}"

      filename = download(url)

      store_file(to_bucket,to_key,filename)

      File.delete(filename)

      return "http://s3.amazonaws.com/#{to_bucket}/#{to_key}"

    else

      puts "#{from_bucket}.#{from_key} didn't exist"

      return nil

    end

  end



  def store_file(bucket, key, filename)

     puts "Storing #{filename} in #{bucket}.#{key}"

     S3Object.store(

      key,

      File.open(filename),

      bucket,

      :access => :public_read

      )

  end



  def download(url, save_as = nil)

    if save_as.nil?

      Dir.mkdir("amazon_s3_temp") if !File.exists?("amazon_s3_temp")

      save_as = File.join("amazon_s3_temp",File.basename(url))

    end

    begin

      puts "Saving #{url} to #{save_as}"

      agent = WWW::Mechanize.new {|a| a.log = Logger.new(STDERR) }

      img = agent.get(url)

      img.save_as(save_as)

      return save_as

    rescue

      raise "Failed on " + url + "  " + save_as

    end

  end



  def exists?(bucket,key)

    begin

      res = S3Object.find key, bucket

    rescue 

      res = nil

    end

    return !res.nil?

  end

      

end

upload_to_s3 - Ruby S3 upload client

peter (Peter Cooperx) — Fri, 01 Jun 2007 17:21:39 GMT

Prerequisites:
gem install aws-s3
gem install main

#!/bin/env ruby



require 'rubygems'

require 'main'

require 'aws/s3'

include AWS::S3



Main {

  argument('source_filename') {

    cast :string

    description 'source filename to copy to S3'

  }



  argument('bucket_name') {

    cast :string

    description 'bucket to place the file in on S3'

  }



  option('access_key_id') {

    argument :optional

    description 'specify the access_key_id manually'

    default 'put your access key here if you want'

  }



  option('secret_access_key') {

    argument :optional

    description 'specify the secret key manually'

    default 'put your secret key here if you want'

  }



  def run

    bucket_name = params['bucket_name'].value

    source_filename = params['source_filename'].value



    Base.establish_connection!(

      :access_key_id     => params['access_key_id'].value,

      :secret_access_key => params['secret_access_key'].value

    )



    begin

      Bucket.find(bucket_name)

    rescue

      puts "Need to make bucket #{bucket_name}.."

      Bucket.create(bucket_name)



      # Confirm its existence..

      Bucket.find(bucket_name)

    end



    puts "Got bucket.."

    puts "Uploading #{File.basename(source_filename)}.."

    S3Object.store(File.basename(source_filename), open(source_filename), bucket_name)

    puts "Stored!"



    exit_success!

  end

}

Ruby Client for Amazon Alexa Site Thumbnail (AST) Service

peter (Peter Cooperx) — Tue, 05 Dec 2006 18:31:27 GMT

It's scrappy, but it does the job.



require 'cgi'

require 'openssl'

require 'base64'

require 'open-uri'



access_id = 'YOUR_ACCESS_ID'

secret_id = 'YOUR_SECRET_ID'



source_url = ARGV.first



timestamp = Time.now.strftime("%Y-%m-%dT%H:%M:%SZ")

sig = Base64.encode64(OpenSSL::HMAC::digest(OpenSSL::Digest::Digest.new('SHA1'), secret_id, 'Thumbnail' + timestamp)).strip



url = "http://ast.amazonaws.com/Xino?Action=Thumbnail&AWSAccessKeyId=" + access_id

url << "&Signature=" + CGI.escape(sig)

url << "&Timestamp=" + CGI.escape(timestamp)

url << "&Url=" +  source_url



begin

  doc = open(url).read

rescue

  puts "Could not access AWS"

  exit

end



m = doc.match(/\]+exists=\"true\"\>(.+?)\<\//i)



if m && m[1]

  thumb_url = m[1]

  thumb_url.gsub!(/\&/, '&')

  File.open("#{source_url}.jpg", "w") { |f| f.write open(thumb_url).read }

  puts "Saved to #{source_url}.jpg"

elsif m && m.match(/exists=\"false\"/)

  puts "No thumbnail for #{source_url}"

else

  puts "Error"

end

S3 upload client for Ruby

peter (Peter Cooperx) — Thu, 30 Nov 2006 10:27:44 GMT

Uses Marcel Molina's AWS::S3 gem.. gem install aws-s3



#!/usr/bin/env ruby



require 'rubygems'

require 'aws/s3'



local_file = ARGV[0]

bucket = ARGV[1]

mime_type = ARGV[2] || "application/octet-stream"



AWS::S3::Base.establish_connection!(

  :access_key_id     => 'REPLACE_ME',

  :secret_access_key => 'REPLACE_ME'

)



base_name = File.basename(local_file)



puts "Uploading #{local_file} as '#{base_name}' to '#{bucket}'"



AWS::S3::S3Object.store(

  base_name,

  File.open(local_file),

  bucket,

  :content_type => mime_type

)



puts "Uploaded!"

Amazon REST API for newLISP

frontera000 (bob bae) — Fri, 13 Oct 2006 10:16:27 GMT



;; (amazon-op "Operation=ItemSearch&SearchIndex=Books&ItemPage=1&Keywords=Cloud+Atlas&Respon\

seGroup=Request,Small")

;;

;; (amazon-op "Operation=SimilarityLookup&ItemId=1400063795,0812966929&SimilarityType=Random\

&ResponseGroup=Request,Large")

;; (amazon-op "Operation=SimilarityLookup&ItemId=1400063795,061873516X,0812966929&Similarity\

Type=Random&ResponseGroup=Request,Small")



;;  (setq x (xml-digest (amazon-op "Operation=ItemLookup&ItemId=0375507256&ResponseGroup=Req\

uest,Similarities,ListmaniaLists")))

;; Using x get a list of listmanias for an item

;;  (map (lambda (x) (x 1 2 2)) (1 -1 (x (chop (ref 'ListmaniaLists  x)))))



;; (setq y (amazon-op "Operation=ListLookup&ListType=Listmania&ListId=R21LV6VJEZ794O&Respons\

eGroup=ListFull"))

;; Using y get a list of ISBNs from a listmania

;; (map (lambda(x) (x 4 1 2 )) (8 -1 ( (y (chop (ref 'Lists y)) ) 2))) ; gives list of ISBNs\

 from the listmania



;; get a list of listmania details from a list of listmanias.

;;  (setq z   (map (lambda (x) (xml-digest (amazon-op (append "Operation=ListLookup&ListType\

=Listmania&ListId=" x "&ResponseGroup=ListFull"))))  (map (lambda (x) (x 1 2 2)) (1 -1 (x (c\

hop (ref 'ListmaniaLists  x))))))  )

;;

;; get a list of ISBNs from all of the listmanias for the item

;; (setq z (flat (map (lambda(y) (map (lambda(x) (x 4 1 2 )) (8 -1 ( (y (chop (ref 'Lists y)\

) ) 2))) ) z)))

;;



;; get item info for all the ISBNs from all listmanias related to the original item

;; (setq w (map (lambda (x) (xml-digest (amazon-op (append "Operation=ItemLookup&ItemId=" x)\

)))  (unique z)))



;; get a list of pair (title,author) from w above.

;; (map (lambda (x) (list ((x (chop (ref 'Title x))) 1)  (if (ref 'Author x)((x (chop (ref '\

Author x)))1) "???"))) w)



(define (amazon-op params)

  (get-url (append 

"http://webservices.amazon.com/onca/xml?Service=AWSECommerceService&&AWSAccessKeyId=YOUROWNKEY&" params)))



(define (xml-digest result)

  (xml-type-tags nil nil nil nil)

  (setq xresult (xml-parse result (+ 1 2 4 8 16))))

library mashup

frontera000 (bob bae) — Sat, 30 Sep 2006 03:52:13 GMT

// description of your code here



;; combine APIs and other web available bibliographic data.

;; Use amazon, LibraryThing, and OCLC Worldcat.

;; Get ISBN numbers for a book by searching author and title.

;; Get the list of available ISBNs from different sources.

;; Get "tags" available for a book from LibraryThing for

;; a given book.

;; etc.

;;

;; N.B. A person from LibraryThing.com posted a comment

;; regarding this program, indicating that the tag cloud data

;; is copyrighted. 

;;

;; Here's the quoted message:

;;

;; Tim's message: "And it's fine here, as a test, and probably in other 

;; contexts where it could be considered "fair use" under copyright. 

;; But--unlike the thingISBN service and some other LibraryThing 

;; APIs--LibraryThing's tag clouds are not free for public use, 

;; outside of the RSS feeds and widgets we provide. 

;; We're not sure how and under what license to 

;; release them when we do."

;;

;; So, I wrote this for fun.  If I were you I wouldn't run it, just read

;; the code.  :-)

;;





(define (search-worldcat title author)

  (setq title (replace " " title "+"))

  (setq author (replace " " author "+"))

  (get-url (append "http://worldcatlibraries.org/search?q=ti:" title "+au:" author "&qt=advanced")))



(setq oclc-url-1-pattern {

(define (oclc-url-1 str)

  (setq loc (find oclc-url-1-pattern str))

  (setq loc (+ loc (length oclc-url-1-pattern)))

  (setq loc (+ 1 loc))

  (setq loc-end (find ">" (loc -1 str)))

  (setq loc-end (- loc-end 1))

  (setq loc-url (loc loc-end str))

  (println (append "http://worldcatlibraries.org" loc-url))

  (get-url (append "http://worldcatlibraries.org" loc-url)))



(setq oclc-isbn-pattern "ISBN: ")



(define (find-isbn str)

  (setq loc (find oclc-isbn-pattern str))

  (setq loc (+ loc (length oclc-isbn-pattern)))

  (setq loc-end (find "" (loc -1 str)))

  (loc loc-end str))



(define (thing-isbn isbn)

  (xml-type-tags nil nil nil nil)

  (setq isbn-data 

      (xml-parse (get-url  

          (append "http://www.librarything.com/api/thingISBN/" isbn)))))



(define (get-isbn-list isbn-data)

  (setq indexer (ref "isbn" isbn-data))

  (nth-set 2 indexer 2)

  (setq num-isbns (length  (isbn-data 0)))

  (setq isbn-list '())

  (for (idx (first (1 indexer))  (- num-isbns 1))

        (push (isbn-data (first (0 indexer)) idx (first (2 indexer))) isbn-list -1))

   isbn-list)



(define (add-explorer-path)

   (env "PATH" (append (env "PATH" ) ";c:\\program files\\internet explorer")))



(define (show-amazon-isbn isbn)

   (process (append "iexplore " "http://www.amazon.com/exec/obidos/ASIN/"

           isbn )))





(define (get-librarything-isbn isbn)

    (get-url (append "http://librarything.com/isbn/" isbn)))



(define (get-librarything-tagsection str)

    (setq loc (find "Tags used" str ))

    (setq mystr (loc -1 str))

    (setq loc-end (find "
" mystr))

     (0 loc-end mystr))



(define (get-librarything-taglist mystr)

    (setq tags-list '())

    (while (setq tag-loc (find "/tag/" mystr))

        (setq tag-loc-end (find "target=" (tag-loc -1 mystr)))

        (push ((+ tag-loc 5) (- tag-loc-end 7) mystr) tags-list -1)

        (setq mystr ((+ tag-loc tag-loc-end ) -1 mystr)))

    tags-list)



(define (show-librarything-tag tag)

   (process (append "iexplore " "http://www.librarything.com/tag/"

           tag )))

Amazon Statistically Improbable Phrases Search Bookmarklet

sage (Sage) — Sat, 16 Apr 2005 01:58:37 GMT

javascript:q=document.getSelection();for(i=0;i