Never been to DZone Snippets before?

Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

« Newer Snippets
Older Snippets »
Showing 1-9 of 9 total  RSS 

Creating a bucket in Amazon S3 through an irb session

1) Log into an irb session, and enter your S3 login details.
require 'rubygems'
require 'aws/s3'

  AWS::S3::Base.establish_connection!(
    :access_key_id     => 'REPLACE_ME',
    :secret_access_key => 'REPLACE_ME'
  )

output:
=> #<AWS::S3::Connection:0xb75e0594 @http=#<Net::HTTP s3.amazonaws.com:80 open=false>, @secret_access_key="", @options={:server=>"s3.amazonaws.com", :access_key_id=>"", :port=>80, :secret_access_key=>"", :persistent=>true}, @access_key_id="19S45GYAGWK8DC2B8VG2">

2) Browse the existing buckets.
AWS::S3::Service.buckets

output:
=> [#<AWS::S3::Bucket:0xb75cc850 @object_cache=[], @attributes={"name"=>"ogg.twitteraudio.com", "creation_date"=>Sat Apr 26 10:40:16 UTC 2008}>, #<AWS::S3::Bucket:0xb75cc83c @object_cache=[], @attributes={"name"=>"t1000", "creation_date"=>Fri Apr 25 21:35:21 UTC 2008}>, #<AWS::S3::Bucket:0xb75cc814 @object_cache=[], @attributes={"name"=>"t2000", "creation_date"=>Fri Apr 25 21:53:15 UTC 2008}>]

3) Browse the buckets in a programmatical way.
AWS::S3::Service.buckets.each {|b| puts b.name}

output:
ogg.twitteraudio.com
t1000
t2000


4) Add a new bucket called t3000.
AWS::S3::Bucket.create('t3000')

output:
=> true

5) Observe adding the bucket again doesn't cause an error.
AWS::S3::Bucket.create('t3000')

output:
=> true

6) View the buckets again. 
AWS::S3::Service.buckets

output:
=> [#<AWS::S3::Bucket:0xb75cc850 @object_cache=[], @attributes={"name"=>"ogg.twitteraudio.com", "creation_date"=>Sat Apr 26 10:40:16 UTC 2008}>, #<AWS::S3::Bucket:0xb75cc83c @object_cache=[], @attributes={"name"=>"t1000", "creation_date"=>Fri Apr 25 21:35:21 UTC 2008}>, #<AWS::S3::Bucket:0xb75cc814 @object_cache=[], @attributes={"name"=>"t2000", "creation_date"=>Fri Apr 25 21:53:15 UTC 2008}>]

Note: You would expect t3000 to be in there however it didn't appear possibly because of the bucket permissions.

7) Let's then look for bucket t3000.
t3000 = AWS::S3::Bucket.find('t3000')

output:
=> #<AWS::S3::Bucket:0xb76df724 @object_cache=[], @attributes={"prefix"=>nil, "name"=>"t3000", "marker"=>nil, "max_keys"=>1000, "is_truncated"=>false, "xmlns"=>"http://s3.amazonaws.com/doc/2006-03-01/"}>

8) Now that we've found the bucket let's upload a text file called works.txt.
file = "works.txt"

output:
=> "works.txt"
AWS::S3::S3Object.store(file, open(file), 't3000', :access => :public_read)

output:
=> #<AWS::S3::S3Object::Response:0x-608926458 200 OK>

9) Setting the file access to :public_read allows us to view the file from the http location http://t3000.s3.amazonaws.com/works.txt

References:
http://amazon.rubyforge.org/
upload_to_s3 - Ruby S3 upload client [dzone.com]

*update: 14:30 30 April 2008 *
I didn't use Bucket.objects(:reload) which is the reason why the bucket t3000 didn't show up with the statement Service.buckets

Reference: spatten design - Amazon S3, Ruby and Rails slides [spattendesign.com]
to ruby find Amazon http service store s3 bucket buckets by jrobertson on Apr 29, 2008

Using ruby to upload multiple files to Amazon S3

This ruby code builds upon the code S3 upload client for Ruby [dzone.com] by reading the account details and file details from XML files.

#!/usr/bin/env ruby
#file : rubys3file2upload.rb

require 'rubygems'
require 'aws/s3'
require 'rexml/document'
include REXML

class S3FileUpload

  def initialize(xml_accounts_file, xml_upload_file)
    initialize_account(xml_accounts_file)
    main(xml_upload_file)
  end

  def initialize_account(accounts_file)

    file = File.new(accounts_file)
    doc = Document.new(file)

    account = doc.root.elements['records/access']
    h = Hash.new
    h[:access_key_id] = account.elements['key_id'].text.to_s
    h[:secret_access_key] = account.elements['secret'].text.to_s
    AWS::S3::Base.establish_connection!(h)

  end

  def main(xml_upload_file)
    file = File.new(xml_upload_file)
    doc = Document.new(file)
puts doc
    doc.root.elements.each('records/file') do |f|
      local_file = f.elements['local'].text.to_s
      bucket = f.elements['bucket'].text.to_s
      mime_type = f.elements['mime_type'].text.to_s
      upload(local_file, mime_type, bucket)
    end
  end

  def upload(local_file, mime_type, bucket)

    base_name = File.basename(local_file)

    puts "Uploading #{local_file} as '#{base_name}' to '#{bucket}'"

    AWS::S3::S3Object.store(
      base_name,
      File.open(local_file),
      bucket,
      :content_type => mime_type,
      :access => :public_read
    )

    puts "Uploaded!"
  end

end

if __FILE__ == $0
  s3fu = S3FileUpload.new('s3accounts.xml', 's3files2upload.xml')
end


file: s3accounts.xml
<s3accounts>
  <summary/>
  <records>
    <access id="100"><name>REPLACE_ME</name><key_id>REPLACE_ME</key_id><secret>REPLACE_ME</secret></access>
  </records>
</s3accounts>


file: s3files2upload.xml
<s3files2upload>
  <summary/>
  <records>
    <file>
      <local>autocomplete.html</local>
      <bucket>t2000</bucket>
      <mime_type>text/html</mime_type>
    </file>
    <file>
      <local>autocomplete2.html</local>
      <bucket>t2000</bucket>
      <mime_type>text/html</mime_type>
    </file>
    <file>
      <local>autocomplete3.html</local>
      <bucket>t2000</bucket>
      <mime_type>text/html</mime_type>
    </file>
  </records>
</s3files2upload>


Note: The file given the correct permission could be read from http://t2000.s3.amazonaws.com/autocomplete.html

Reference: http://amazon.rubyforge.org/
to ruby xml Amazon rexml s3 account by jrobertson on Apr 26, 2008

Simple S3 utils - copy bucket to bucket

Some simple S3 utils in Ruby

Specifically written to copy one bucket to another (for testing on production on staging)

PLEASE BE CAREFUL WITH THIS - THERE IS CODE THAT DELETES ALL CONTENTS OF A BUCKET

Example:
a = AmazoneS3Asset.new
a.copy_over_bucket("myapp_production", "myapp_production")



require 'aws/s3'
require 'mechanize'

class AmazonS3Asset
  
  include AWS::S3
  S3ID = "your s3 id"
  S3KEY = "your s3 key"
  
  def initialize
    puts "connecting..."
    AWS::S3::Base.establish_connection!(
      :access_key_id     => S3ID,
      :secret_access_key => S3KEY
    )
  end

  def delete_key(bucket, key)
    if exists?(bucket, key) 
      S3Object.delete key, bucket
    end
  end
  
  def empty_bucket(bucket)
    bucket_keys(bucket).each do |k|
      puts "deleting #{k}"
      delete_key(bucket,k)
    end
  end
  
  def bucket_keys(bucket)
    b = Bucket.find(bucket)
    b.objects.collect {|o| o.key}
  end

  def copy_over_bucket(from_bucket, to_bucket)
    puts "Replacing #{to_bucket} with contents of #{from_bucket}"
    #delete to_bucket
    empty_bucket(to_bucket)
    bucket_keys(from_bucket).each do |k|
      copy_between_buckets(from_bucket, to_bucket, k)
    end
  end
  
  def copy_between_buckets(from_bucket, to_bucket, from_key, to_key = nil)
    if exists?(from_bucket, from_key)
      to_key = from_key if to_key.nil?
      puts "Copying #{from_bucket}.#{from_key} to #{to_bucket}.#{to_key}"
      url = "http://s3.amazonaws.com/#{from_bucket}/#{from_key}"
      filename = download(url)
      store_file(to_bucket,to_key,filename)
      File.delete(filename)
      return "http://s3.amazonaws.com/#{to_bucket}/#{to_key}"
    else
      puts "#{from_bucket}.#{from_key} didn't exist"
      return nil
    end
  end

  def store_file(bucket, key, filename)
     puts "Storing #{filename} in #{bucket}.#{key}"
     S3Object.store(
      key,
      File.open(filename),
      bucket,
      :access => :public_read
      )
  end

  def download(url, save_as = nil)
    if save_as.nil?
      Dir.mkdir("amazon_s3_temp") if !File.exists?("amazon_s3_temp")
      save_as = File.join("amazon_s3_temp",File.basename(url))
    end
    begin
      puts "Saving #{url} to #{save_as}"
      agent = WWW::Mechanize.new {|a| a.log = Logger.new(STDERR) }
      img = agent.get(url)
      img.save_as(save_as)
      return save_as
    rescue
      raise "Failed on " + url + "  " + save_as
    end
  end

  def exists?(bucket,key)
    begin
      res = S3Object.find key, bucket
    rescue 
      res = nil
    end
    return !res.nil?
  end
      
end
to ruby copy Amazon s3 util bucket buckets by wiseleyb on Dec 30, 2007

upload_to_s3 - Ruby S3 upload client

Prerequisites:
gem install aws-s3
gem install main

#!/bin/env ruby

require 'rubygems'
require 'main'
require 'aws/s3'
include AWS::S3

Main {
  argument('source_filename') {
    cast :string
    description 'source filename to copy to S3'
  }

  argument('bucket_name') {
    cast :string
    description 'bucket to place the file in on S3'
  }

  option('access_key_id') {
    argument :optional
    description 'specify the access_key_id manually'
    default 'put your access key here if you want'
  }

  option('secret_access_key') {
    argument :optional
    description 'specify the secret key manually'
    default 'put your secret key here if you want'
  }

  def run
    bucket_name = params['bucket_name'].value
    source_filename = params['source_filename'].value

    Base.establish_connection!(
      :access_key_id     => params['access_key_id'].value,
      :secret_access_key => params['secret_access_key'].value
    )

    begin
      Bucket.find(bucket_name)
    rescue
      puts "Need to make bucket #{bucket_name}.."
      Bucket.create(bucket_name)

      # Confirm its existence..
      Bucket.find(bucket_name)
    end

    puts "Got bucket.."
    puts "Uploading #{File.basename(source_filename)}.."
    S3Object.store(File.basename(source_filename), open(source_filename), bucket_name)
    puts "Stored!"

    exit_success!
  end
}
to ruby Amazon backup upload s3 aws storage by peter on Jun 01, 2007

Ruby Client for Amazon Alexa Site Thumbnail (AST) Service

It's scrappy, but it does the job.

require 'cgi'
require 'openssl'
require 'base64'
require 'open-uri'

access_id = 'YOUR_ACCESS_ID'
secret_id = 'YOUR_SECRET_ID'

source_url = ARGV.first

timestamp = Time.now.strftime("%Y-%m-%dT%H:%M:%SZ")
sig = Base64.encode64(OpenSSL::HMAC::digest(OpenSSL::Digest::Digest.new('SHA1'), secret_id, 'Thumbnail' + timestamp)).strip

url = "http://ast.amazonaws.com/Xino?Action=Thumbnail&AWSAccessKeyId=" + access_id
url << "&Signature=" + CGI.escape(sig)
url << "&Timestamp=" + CGI.escape(timestamp)
url << "&Url=" +  source_url

begin
  doc = open(url).read
rescue
  puts "Could not access AWS"
  exit
end

m = doc.match(/\<aws:thumbnail[^\>]+exists=\"true\"\>(.+?)\<\//i)

if m && m[1]
  thumb_url = m[1]
  thumb_url.gsub!(/\&amp;/, '&')
  File.open("#{source_url}.jpg", "w") { |f| f.write open(thumb_url).read }
  puts "Saved to #{source_url}.jpg"
elsif m && m.match(/exists=\"false\"/)
  puts "No thumbnail for #{source_url}"
else
  puts "Error"
end
to ruby Amazon screenshot screenshots thumbnail aws ast alexa by peter on Dec 05, 2006 3 Comments

S3 upload client for Ruby

Uses Marcel Molina's AWS::S3 gem.. gem install aws-s3

#!/usr/bin/env ruby

require 'rubygems'
require 'aws/s3'

local_file = ARGV[0]
bucket = ARGV[1]
mime_type = ARGV[2] || "application/octet-stream"

AWS::S3::Base.establish_connection!(
  :access_key_id     => 'REPLACE_ME',
  :secret_access_key => 'REPLACE_ME'
)

base_name = File.basename(local_file)

puts "Uploading #{local_file} as '#{base_name}' to '#{bucket}'"

AWS::S3::S3Object.store(
  base_name,
  File.open(local_file),
  bucket,
  :content_type => mime_type
)

puts "Uploaded!"
to ruby Amazon s3 aws by peter on Nov 30, 2006 2 Comments

Amazon REST API for newLISP

;; (amazon-op "Operation=ItemSearch&SearchIndex=Books&ItemPage=1&Keywords=Cloud+Atlas&Respon\
seGroup=Request,Small")
;;
;; (amazon-op "Operation=SimilarityLookup&ItemId=1400063795,0812966929&SimilarityType=Random\
&ResponseGroup=Request,Large")
;; (amazon-op "Operation=SimilarityLookup&ItemId=1400063795,061873516X,0812966929&Similarity\
Type=Random&ResponseGroup=Request,Small")

;;  (setq x (xml-digest (amazon-op "Operation=ItemLookup&ItemId=0375507256&ResponseGroup=Req\
uest,Similarities,ListmaniaLists")))
;; Using x get a list of listmanias for an item
;;  (map (lambda (x) (x 1 2 2)) (1 -1 (x (chop (ref 'ListmaniaLists  x)))))

;; (setq y (amazon-op "Operation=ListLookup&ListType=Listmania&ListId=R21LV6VJEZ794O&Respons\
eGroup=ListFull"))
;; Using y get a list of ISBNs from a listmania
;; (map (lambda(x) (x 4 1 2 )) (8 -1 ( (y (chop (ref 'Lists y)) ) 2))) ; gives list of ISBNs\
 from the listmania

;; get a list of listmania details from a list of listmanias.
;;  (setq z   (map (lambda (x) (xml-digest (amazon-op (append "Operation=ListLookup&ListType\
=Listmania&ListId=" x "&ResponseGroup=ListFull"))))  (map (lambda (x) (x 1 2 2)) (1 -1 (x (c\
hop (ref 'ListmaniaLists  x))))))  )
;;
;; get a list of ISBNs from all of the listmanias for the item
;; (setq z (flat (map (lambda(y) (map (lambda(x) (x 4 1 2 )) (8 -1 ( (y (chop (ref 'Lists y)\
) ) 2))) ) z)))
;;

;; get item info for all the ISBNs from all listmanias related to the original item
;; (setq w (map (lambda (x) (xml-digest (amazon-op (append "Operation=ItemLookup&ItemId=" x)\
)))  (unique z)))

;; get a list of pair (title,author) from w above.
;; (map (lambda (x) (list ((x (chop (ref 'Title x))) 1)  (if (ref 'Author x)((x (chop (ref '\
Author x)))1) "???"))) w)

(define (amazon-op params)
  (get-url (append 
"http://webservices.amazon.com/onca/xml?Service=AWSECommerceService&&AWSAccessKeyId=YOUROWNKEY&" params)))

(define (xml-digest result)
  (xml-type-tags nil nil nil nil)
  (setq xresult (xml-parse result (+ 1 2 4 8 16))))
to xml Amazon rest lisp api newlisp by frontera000 on Oct 13, 2006

library mashup

// description of your code here

;; combine APIs and other web available bibliographic data.
;; Use amazon, LibraryThing, and OCLC Worldcat.
;; Get ISBN numbers for a book by searching author and title.
;; Get the list of available ISBNs from different sources.
;; Get "tags" available for a book from LibraryThing for
;; a given book.
;; etc.
;;
;; N.B. A person from LibraryThing.com posted a comment
;; regarding this program, indicating that the tag cloud data
;; is copyrighted. 
;;
;; Here's the quoted message:
;;
;; Tim's message: "And it's fine here, as a test, and probably in other 
;; contexts where it could be considered "fair use" under copyright. 
;; But--unlike the thingISBN service and some other LibraryThing 
;; APIs--LibraryThing's tag clouds are not free for public use, 
;; outside of the RSS feeds and widgets we provide. 
;; We're not sure how and under what license to 
;; release them when we do."
;;
;; So, I wrote this for fun.  If I were you I wouldn't run it, just read
;; the code.  :-)
;;


(define (search-worldcat title author)
  (setq title (replace " " title "+"))
  (setq author (replace " " author "+"))
  (get-url (append "http://worldcatlibraries.org/search?q=ti:" title "+au:" author "&qt=advanced")))

(setq oclc-url-1-pattern {<div class="name"><a href=})

(define (oclc-url-1 str)
  (setq loc (find oclc-url-1-pattern str))
  (setq loc (+ loc (length oclc-url-1-pattern)))
  (setq loc (+ 1 loc))
  (setq loc-end (find ">" (loc -1 str)))
  (setq loc-end (- loc-end 1))
  (setq loc-url (loc loc-end str))
  (println (append "http://worldcatlibraries.org" loc-url))
  (get-url (append "http://worldcatlibraries.org" loc-url)))

(setq oclc-isbn-pattern "<strong>ISBN: </strong>")

(define (find-isbn str)
  (setq loc (find oclc-isbn-pattern str))
  (setq loc (+ loc (length oclc-isbn-pattern)))
  (setq loc-end (find "</li>" (loc -1 str)))
  (loc loc-end str))

(define (thing-isbn isbn)
  (xml-type-tags nil nil nil nil)
  (setq isbn-data 
      (xml-parse (get-url  
          (append "http://www.librarything.com/api/thingISBN/" isbn)))))

(define (get-isbn-list isbn-data)
  (setq indexer (ref "isbn" isbn-data))
  (nth-set 2 indexer 2)
  (setq num-isbns (length  (isbn-data 0)))
  (setq isbn-list '())
  (for (idx (first (1 indexer))  (- num-isbns 1))
        (push (isbn-data (first (0 indexer)) idx (first (2 indexer))) isbn-list -1))
   isbn-list)

(define (add-explorer-path)
   (env "PATH" (append (env "PATH" ) ";c:\\program files\\internet explorer")))

(define (show-amazon-isbn isbn)
   (process (append "iexplore " "http://www.amazon.com/exec/obidos/ASIN/"
           isbn )))


(define (get-librarything-isbn isbn)
    (get-url (append "http://librarything.com/isbn/" isbn)))

(define (get-librarything-tagsection str)
    (setq loc (find "Tags used" str ))
    (setq mystr (loc -1 str))
    (setq loc-end (find "</div>" mystr))
     (0 loc-end mystr))

(define (get-librarything-taglist mystr)
    (setq tags-list '())
    (while (setq tag-loc (find "/tag/" mystr))
        (setq tag-loc-end (find "target=" (tag-loc -1 mystr)))
        (push ((+ tag-loc 5) (- tag-loc-end 7) mystr) tags-list -1)
        (setq mystr ((+ tag-loc tag-loc-end ) -1 mystr)))
    tags-list)

(define (show-librarything-tag tag)
   (process (append "iexplore " "http://www.librarything.com/tag/"
           tag )))
to Amazon lisp api oclc librarything worldcat by frontera000 on Sep 29, 2006

Amazon Statistically Improbable Phrases Search Bookmarklet

javascript:q=document.getSelection();for(i=0;i<frames.length;i++){q=frames[i].document.getSelection();if(q)break;}if(!q)void(q=prompt('Statistically Improbable Phrase:',''));if(q)location.href='http://www.amazon.com/gp/phrase/'+escape(q)+'/ref=sip_bod_0/103-6410487-8187016'
to bookmarklet Amazon by sage on Apr 15, 2005
« Newer Snippets
Older Snippets »
Showing 1-9 of 9 total  RSS