I often use rsync to keep a local copy of some HTTPD logs (around ~200MB atm.). Since they are append-only, having rsync compute and compare the checksums for the parts I already have seems wasteful: both my box and the one I'm copying from would be happier if they didn't have to process a couple hundred MBs for nothing. (...)

   1  
   2  #!/usr/bin/env ruby
   3  
   4  REMOTE_RUBY = "ruby"
   5  # TODO: allow REMOTE_RUBY to be specified via a cmdline opt
   6  
   7  if ARGV.size != 2 || ARGV[0][/:/].nil? || !File.exist?(ARGV[1])
   8    puts <<EOF
   9    ruby logfetcher.rb host:path/to/src dst
  10  EOF
  11    exit
  12  end
  13  
  14  FILE = ARGV[1]
  15  REMOTE_HOST, REMOTE_FILE = ARGV[0].split(/:/)
  16  BLOCK_SIZE = 8192
  17  
  18  osize = File.size(FILE)
  19  #FIXME: cheap escaping
  20  command = "File.open(#{REMOTE_FILE.inspect}){|f| " + 
  21            "f.pos = #{osize}; print f.read(#{BLOCK_SIZE}) until f.eof? }"
  22  
  23  command.gsub!(/"/){'\\"'}
  24  fetched = 0
  25  t = nil
  26  $stdout.sync = true
  27  print "Establishing connection\r"
  28  File.open(FILE, "a") do |os|
  29    IO.popen(%{ssh #{REMOTE_HOST} ruby -e '"#{command}"'}) do |is|
  30      until is.eof?
  31        data = is.read(BLOCK_SIZE)
  32        t ||= Time.new # ignore the time it takes to establish the SSH connection
  33        fetched += data.size
  34        print "Read #{fetched}                          \r"
  35        os.write(data)
  36      end
  37    end
  38  end
  39  print(" " * 50  + "\r")
  40  
  41  dt = Time.new - t
  42  puts "Fetched #{fetched} bytes."
  43  puts "Total size #{osize + fetched}."
  44  puts "Needed %4.1f seconds." % dt
  45  puts "Average speed %d bytes/sec." % (fetched / dt)

Source: Cheapest rsync replacement