Never been to DZone Snippets before?

Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

« Newer Snippets
Older Snippets »
Showing 1-10 of 19 total  RSS 

Restore a single table from a large MySQL backup

Say, for some reason, you need to restore the entire contents of a single table from a HUGE mysqldump generated backup containing several tables. For example:

create table `baz`;

GIGS OF SQL YOU DON'T WANT;

create table `foo`;

A COUPLE THOUSAND LINES YOU DO WANT;

create table `bar`;

MORE SQL YOU DON'T WANT;


With a little dash 'o ruby, you can extract just the part you want:

$ ruby -ne '@found=true if $_ =~ /^CREATE TABLE `foo`/i; next unless @found; exit if $_ =~ /^CREATE TABLE (?!`foo`)/i; puts $_;' giant_sql_dump.sql > foo.sql
$ cat foo.sql
create table `foo`;

A COUPLE THOUSAND LINES YOU DO WANT;



You can then easily restore that entire table:

$ mysql mydatabase -e 'drop table foo'
$ mysql mydatabase < foo.sql

Active Record YAML Backup Solution - Drop in rake task and config file to backup db and directorys of your choice (user uploaded files)

EZ drop in backup rake task for your rails projects - works well - 5 - 10 min set up, one rake file and one config file
backs up db and any directory's to any number of servers with rsync


BSD License or whatever, but it would be cool if you told me your using it
2007 ISS http://industrialstrengthinc.com

this is a sample config file and a rake task you can drop into any rails project to do backups, easy to automate.
Backs up your specified environment db to activerecord yml files (one per table) and zips them up in a human readable timestamped file
syncs that and any other set of arbitrary directory's to any number of arbitrary servers with rsync
be sure to set ssh key based logins
to run: rake backup
also: rake backup:db, rake backup:restoredb (promts for a file created by backup:db), rake backup:push (to push out what u got)
note: this uses something like 30 lines of code from some blog site I got it from that I cant recall, yay for that guy, thanks

## example crontab entry:

3 * * * * cd /rails_deployment_dir/current && nice rake backup RAILS_ENV=production >> /rails_deployment_dir/production_backup_system.log

## config file - goes in config/backup.yml

production: 
  dirs: 
   - db/backups
   - public/uploaded_images
  servers: 
    -  name: backup server number 1
       host: gridserver.com
       port: 22
       user: blah@whatever.com
       dir: /home/blah/backups
    -  name: backup server two
       host: kradradio.com
       port: 22
       user: kraduser
       dir: /home/kraduser/backups_from_my_rails_proj


development: 
  dirs: 
   - db/backups
   - public/uploaded_images
  servers: 
    -  name: local self
       host: localhost
       port: 5222
       user: oneman
       dir: /home/oneman/Documents/development_backup


## rake file lib/tasks/backup.rake

desc "Backup Everything Specified in config/backup.yml"
task :backup => [ "backup:db",  "backup:push"]

namespace :backup do
 
    RAILS_APPDIR = RAILS_ROOT.sub("/config/..","")
    
   def interesting_tables
     ActiveRecord::Base.connection.tables.sort.reject! do |tbl|
       ['schema_info', 'sessions', 'public_exceptions'].include?(tbl)
     end
   end
  
   desc "Push backup to remote server"
   task :push  => [:environment] do 
      FileUtils.chdir(RAILS_APPDIR)
      backup_config = YAML::load( File.open( 'config/backup.yml' ) )[RAILS_ENV]
      for server in backup_config["servers"]
       puts "Backing up #{RAILS_ENV} directorys #{backup_config['dirs'].join(', ')} to #{server['name']}"
       puts "Time is " + Time.now.rfc2822 + "\n\n"
         for dir in backup_config["dirs"]
          local_dir = RAILS_APPDIR + "/" + dir + "/"
          remote_dir = server['dir'] + "/" + dir.split("/").last + "/"
          puts "Syncing #{local_dir} to #{server['host']}#{remote_dir}"
          sh "/usr/bin/rsync -avz -e 'ssh -p#{server['port']} ' #{local_dir} #{server['user']}@#{server['host']}:#{remote_dir}"
         end
       puts "Completed backup to #{server['name']}\n\n"
      end
   end

    task :storedb => :environment do 

      backupdir = RAILS_APPDIR + '/db/backup'
      FileUtils.mkdir_p(backupdir)
      FileUtils.chdir(backupdir)
      puts "Dumping database to activerecord yaml files in #{backupdir}"
      interesting_tables.each do |tbl|

        klass = tbl.classify.constantize
        puts "Writing #{tbl}..."
        File.open("#{tbl}.yml", 'w+') { |f| YAML.dump klass.find(:all).collect(&:attributes), f }      
      end
      puts "Database Dumped.\n\n"
    end

    desc "Dump Current Environment Db to file"    
    task :db => [:environment, :storedb ] do
      backupdir = RAILS_APPDIR + '/db/backup'
      archivedir = RAILS_APPDIR + '/db/backups'
      backup_filename = "#{RAILS_ENV}_db_backup_#{Time.now.strftime("%B.%d.%Y_at_%I.%M.%S%p_%Z")}.tar.bz2"
      FileUtils.mkdir_p(archivedir)
      puts "Archiving #{backupdir} yaml files to #{backup_filename}\n\n"
      `tar -C #{backupdir} -cjf #{backup_filename} *`
      `mv #{backup_filename} #{archivedir}`
    end

    desc "Restore Current Environment Db from a file"    
    task :restoredb => [:environment] do 
        backupdir = RAILS_APPDIR + '/db/backup'
        archivedir = RAILS_APPDIR + '/db/backups'
        print "Input a file to load into the db: #{archivedir}/"
        backup_filename = STDIN.gets.chomp
        puts "Loading backup file: #{backup_filename}"
        FileUtils.chdir(archivedir)
        `tar -xjf #{backup_filename}`
        `mv *.yml #{backupdir}`
        FileUtils.mkdir_p(backupdir)
        FileUtils.chdir(backupdir)
    
        interesting_tables.each do |tbl|
        puts "Clearing #{tbl} table.."
        ActiveRecord::Base.connection.execute "TRUNCATE #{tbl}"
        puts "Loading #{tbl} backup file..."
        table = YAML.load_file("#{tbl}.yml")        

        if table.length > 0 && table.first.key?("id")
            highestid = 0
            table.each do |fixture|
             if fixture["id"] > highestid
                highestid = fixture["id"]
             end
            end

            ActiveRecord::Base.connection.execute "SELECT setval('#{tbl}_id_seq',#{highestid})"
            puts "Setting #{tbl}_id sequence to #{highestid}"
        end
         
        #klass = tbl.classify.constantize
        ActiveRecord::Base.transaction do 
        
          puts "Inserting #{table.length} values into #{tbl}"
          table.each do |fixture|
            ActiveRecord::Base.connection.execute "INSERT INTO #{tbl} (#{fixture.keys.join(",")}) VALUES (#{fixture.values.collect { |value| ActiveRecord::Base.connection.quote(value) }.join(",")})", 'Fixture Insert'
          end        
          puts "#{tbl} table restored.\n\n"
        end
       end
    end

 
end

Python Flickr Backup Script (Untested)


# flickrbackup.py - Matt Croydon - http://postneo.com
import flickr # http://jamesclarke.info/projects/flickr/
import BitBucket # http://www.other10percent.com/?p=15
import urllib
me = flickr.people_findByUsername("postneo")
bucket = BitBucket.BitBucket("postneo-flickr")
page = 1
total_photos = found_photos = 0
while 1:
    try:
        photos = flickr.people_getPublicPhotos(me.id, 100, page)
        for photo in photos:
            total_photos = total_photos + 1
            if bucket.has_key("%s-%s" % (photo.title, photo.id)):
                pass # we already have this photo
            else:
                data = urllib.urlretrieve("http://static.flickr.com/%s/%s_%s_o.jpg" % (photo.server, photo.id, photo.secret), "flickr.jpg")
                bits = BitBucket.Bits(filename="flickr.jpg")
                bucket["%s-%s" % (photo.title, photo.id)] = bits
                print "saving %s" % photo.title
                found_photos = found_photos + 1
        page = page + 1
    except AttributeError:
        break # We probably got an empty bucket
print "Found %s photos, saved %s new photos" % (total_photos, found_photos)

simple backup PHP application (php files + mysql db)

// backup a folder contains all .php files and it's mysql database. a backup.ini file is required for storing mysql root login and what folder to be backed up.

#!/bin/sh

########################################
# simple sctipt for daily / hourly backup of PHP app + Mysql DB
# Copyleft (c) Sayid Munawar. chenull@yahoo.com
# LICENSE: anyone can copy / modify / distribute. whatever lah
#
# example of workhours backup (8 am - 5 pm. Mon - Fri)
# 0 8-17 * * 1-5 /home/backup/backup.sh
#
# example of simple dayly backup ( 6 pm. Mon - Fri)
# 0 18 * * 1-5 /home/backup/backup.sh
########################################

########################################
# example of backup.ini. REMOVE ALL leading '#'s
# file must be chmod 600
#[main]
#target = /home/backup/storage
#mysql_user = root   ; root can connect to any db
#mysql_pass = secret ; mysql root password
#
#[hukum]
#path = /home/hukum
#mysql_db = hukum
#
#[phpmyadmin]
#path = /home/phpmyadmin
#mysql_db = test
########################################

PATH=$PATH:/usr/bin:/bin:/usr/local/bin

inifile=`dirname $0`/backup.ini

test ! -r $inifile && echo "ERROR! backup.ini not found nor readable" && exit 1

#test apakah backup.ini bisa dibaca orang lain
echo -n `stat -c '%a' $inifile` | grep -q '[0-7]00' >/dev/null 2>&1
test $? -gt 0 && echo "ERROR! $inifile can be read by others" && exit 2
#test `stat -c '%U' $inifile` != 'root' && echo "ERROR! $inifile ownernya bukan root" && exit 3

# function to parse ini file (backup.ini)
#
# $1 -> file.ini
# $2 -> section
_parse_ini () {
    if [ -z "$1" ] || [ -z "$2" ]; then return 0; fi
    eval `cat $1 | \
       sed -e 's/[[:space:]]*\=[[:space:]]*/=/g' \
           -e 's/;.*$//' \
           -e 's/[[:space:]]*$//' \
           -e 's/^[[:space:]]*//' \
           -e "s/^\(.*\)=\([^\"']*\)$/\1=\"\2\"/" | \
       sed -n -e "/^\[$2\]/,/^\s*\[/{/^[^;].*\=.*/p;}"`
}

_parse_ini $inifile main

# coba konek 
mysql -u $mysql_user  -p${mysql_pass} -e '' > /dev/null 2>&1
test $? -gt 0 && echo "ERROR! Failed to connect to mysql" && exit 4

# bikin target kalo blum ada
test ! -d $target && mkdir -p $target

hari=`date +%A`
jam=`date +%H`

# get pwd
cwd=`pwd`

# looping
for section in `cat $inifile | grep '^\[' | grep -v main | sed 's/\[//' | sed 's/\]//'`; do
  outdir=$target/$section/$hari/$jam
  echo Backing up ${section} to $outdir...
  # parse .ini
  _parse_ini $inifile $section
  # cek dulu
  test ! -d $path && echo "Warning: $path not found.  Backup process of $section skipped" && echo && continue
  mysql -u $mysql_user  -p${mysql_pass} -e '' $mysql_db > /dev/null 2>&1
  test $? -gt 0 && echo "Warning: Database $mysql_db not found.  Backup process of $section skipped" && echo && continue

  test ! -d $outdir && mkdir -p $outdir
  cd $path
  tar -czpf $outdir/$section.tar.gz .
  cd $cwd
  mysqldump -Q -u $mysql_user -p${mysql_pass} $mysql_db | gzip > $outdir/${mysql_db}.sql.gz
  ln -fs $outdir/$section.tar.gz $target/$section/latest.tar.gz
  ln -fs $outdir/${mysql_db}.sql.gz $target/$section/latest.sql.gz
done

Backup all your del.icio.us bookmarks to XML

https://api.del.icio.us/v1/posts/all
curl --user accountname:password -o myDelicious.xml -O 'https://api.del.icio.us/v1/posts/all'

Simple mySQL backup script for cron

Simple mySQL backup script for cron - backs up all databases, saves the last 4 copies.
#!/bin/bash

# modify the following to suit your environment
export DB_BACKUP="/backup/mysql_backup"
export DB_USER="root"
export DB_PASSWD="********"

# title and version
echo ""
echo "mySQL_backup"
echo "----------------------"
echo "* Rotating backups..."
rm -rf $DB_BACKUP/04
mv $DB_BACKUP/03 $DB_BACKUP/04
mv $DB_BACKUP/02 $DB_BACKUP/03
mv $DB_BACKUP/01 $DB_BACKUP/02
mkdir $DB_BACKUP/01 

echo "* Creating new backup..."
mysqldump --user=$DB_USER --password=$DB_PASSWD --all-databases | bzip2 > $DB_BACKUP/01/mysql-`date +%Y-%m-%d`.bz2
echo "----------------------"
echo "Done"
exit 0

basic batch drive presence check

// check if removable drives are present. useful with batch backups and letter-jumping flash drives.
// floppies (a:, b:) can cause screen flash asking for floppy to be inserted.
//
// two batch files required - 1. foo.bat
@echo off
echo simple drive check > %temp%\tmp.txt
echo. >> %temp%\tmp.txt
for %%a in (c:, d:, e:, f:, g:) do call bar.bat %%a
cls
type %temp%\tmp.txt
echo.

// 2. bar.bat
%comspec% /f /c vol %1 | find /c "Vol" | choice /c:210
if not errorlevel 2 if errorlevel 1 echo %1 present >> %temp%\tmp.txt
if not errorlevel 3 if errorlevel 2 echo %1 not present >> %temp%\tmp.txt

// screen output can be suppressed by wrapping
ctty nul
for %%a in (c:, d:, e:, f:, g:) do call bar.bat %%a
ctty con

// in foo.bat

upload_to_s3 - Ruby S3 upload client

Prerequisites:
gem install aws-s3
gem install main

#!/bin/env ruby

require 'rubygems'
require 'main'
require 'aws/s3'
include AWS::S3

Main {
  argument('source_filename') {
    cast :string
    description 'source filename to copy to S3'
  }

  argument('bucket_name') {
    cast :string
    description 'bucket to place the file in on S3'
  }

  option('access_key_id') {
    argument :optional
    description 'specify the access_key_id manually'
    default 'put your access key here if you want'
  }

  option('secret_access_key') {
    argument :optional
    description 'specify the secret key manually'
    default 'put your secret key here if you want'
  }

  def run
    bucket_name = params['bucket_name'].value
    source_filename = params['source_filename'].value

    Base.establish_connection!(
      :access_key_id     => params['access_key_id'].value,
      :secret_access_key => params['secret_access_key'].value
    )

    begin
      Bucket.find(bucket_name)
    rescue
      puts "Need to make bucket #{bucket_name}.."
      Bucket.create(bucket_name)

      # Confirm its existence..
      Bucket.find(bucket_name)
    end

    puts "Got bucket.."
    puts "Uploading #{File.basename(source_filename)}.."
    S3Object.store(File.basename(source_filename), open(source_filename), bucket_name)
    puts "Stored!"

    exit_success!
  end
}

Dump postgres production data into development database

When bug requests come in, its great to grab a copy of the current production (or system test env) data and sync it into your development database.

Here's a capistrano recipe for postgresql:

desc "Dumps target database into development db"
task :sync_db do
  env   = ENV['RAILS_ENV'] || ENV['DB'] || 'production'
  file  = "#{application}.sql.bz2"
  remote_file = "#{shared}/log/#{file}"
  run "pg_dump --clean --no-owner --no-privileges -U#{db_user} -h#{db_host} #{db_name}_#{env} | bzip2 > #{file}" do |ch, stream, out|
    ch.send_data "#{db_password}\n" if out =~ /^Password:/
    puts out
  end
  puts rsync = "rsync #{user}@#{domain}:#{file} tmp"
  `#{rsync}`
  puts depackage = "bzcat tmp/#{file} | psql #{local_db_dev}"
  `#{depackage}`
end

rsync backup of a Mac

sudo /usr/local/bin/rsync -aREx --delete --exclude='.Spotlight-*' --exclude '/private/var/vm/*' [IP-address of Mac mini]::PowerBookBackup
« Newer Snippets
Older Snippets »
Showing 1-10 of 19 total  RSS