DZone Snippets: download code

Retrieve your Gmail messages as an XML feed

jrobertson (James Robertson) — Sun, 03 Feb 2008 16:29:06 GMT

This Ruby example shows how to retrieve your most recent email as an atom XML feed from the Google Apps website.



require 'rubygems'

require 'httpclient'



url = "https://mail.google.com/a/yourwebsite.com/feed/atom"

client = HTTPClient.new

client.debug_dev = STDOUT if $DEBUG

client.set_auth(url, 'yourname@yourwebsite.com', 'yourpassword')

resp = client.get(url)

Note: You might require to gem install httpclient.

Download all xkcd.com comics

scvalex (Alexandru Scvortov) — Mon, 15 Oct 2007 18:05:51 GMT

This goes through all the first 329 (you might want to change this) pages, downloading the comic strips.



#!/bin/bash



for i in `seq 1 329`

do

	wget http://xkcd.com/$i/

	wget `grep http://imgs.xkcd.com/comics/ index.html | head -1 | cut -d\" -f2`

	rm index.html

done

Example file download servlet

frost137 (Douglas Wyatt) — Tue, 09 Oct 2007 23:12:04 GMT

// description of your code here



  /**

     *  Sends a file to the ServletResponse output stream.  Typically

     *  you want the browser to receive a different name than the

     *  name the file has been saved in your local database, since

     *  your local names need to be unique.

     *

     *  @param req The request

     *  @param resp The response

     *  @param filename The name of the file you want to download.

     *  @param original_filename The name the browser should receive.

     */

    private void doDownload( HttpServletRequest req, HttpServletResponse resp,

                             String filename, String original_filename )

        throws IOException

    {

        File                f        = new File(filename);

        int                 length   = 0;

        ServletOutputStream op       = resp.getOutputStream();

        ServletContext      context  = getServletConfig().getServletContext();

        String              mimetype = context.getMimeType( filename );



        //

        //  Set the response and go!

        //

        //

        resp.setContentType( (mimetype != null) ? mimetype : "application/octet-stream" );

        resp.setContentLength( (int)f.length() );

        resp.setHeader( "Content-Disposition", "attachment; filename=\"" + original_filename + "\"" );



        //

        //  Stream to the requester.

        //

        byte[] bbuf = new byte[BUFSIZE];

        DataInputStream in = new DataInputStream(new FileInputStream(f));



        while ((in != null) && ((length = in.read(bbuf)) != -1))

        {

            op.write(bbuf,0,length);

        }



        in.close();

        op.flush();

        op.close();

    }

cURL Download

damonp (Damon Parker) — Fri, 27 Apr 2007 04:22:57 GMT

// function to dowload a $remote file and store as $local file, using curl



function curl_download($remote, $local)	{

	$cp = curl_init($remote);

	$fp = fopen($local, "w");

	

	curl_setopt($cp, CURLOPT_FILE, $fp);

	curl_setopt($cp, CURLOPT_HEADER, 0);

	

	curl_exec($cp);

	curl_close($cp);

	fclose($fp);	



}

downloader.py

mcandre (Andrew Pennebaker) — Wed, 25 Oct 2006 07:02:25 GMT



#!/usr/bin/env python



__author__="Andrew Pennebaker (andrew.pennebaker@gmail.com)"

__date__="3 Nov 2005 - 14 Feb 2007"

__copyright__="Copyright 2006 2007 Andrew Pennebaker"

__license__="GPL"

__version__="0.5"

__URL__="http://snippets.dzone.com/posts/show/2887"



from urllib import urlopen



import os



import sys

from getopt import getopt



def getURLName(url):

	directory=os.curdir



	name="%s%s%s" % (

		directory,

		os.sep,

		url.split("/")[-1]

	)



	return name



def createDownload(url, proxy=None):

	instream=urlopen(url, None, proxy)



	filename=instream.info().getheader("Content-Length")

	if filename==None:

		filename="temp"



	return (instream, filename)



def download(instream, outstream):

	outstream.write(instream.read())



	outstream.close()



def usage():

	print "Usage: %s [options] " % (sys.argv[0])

	print "\n--httpproxy "

	print "--ftpproxy "

	print "--gopherproxy "

	print "\n--help (usage)"



	sys.exit()



def main():

	systemArgs=sys.argv[1:] # ignore program name



	urls=[]

	proxies={}



	optlist=[]

	args=[]



	try:

		optlist, args=getopt(systemArgs, None, ["url=", "httpproxy=", "ftpproxy=", "gopherproxy=", "help"])

	except Exception, e:

		usage()



	if len(args)<1:

		usage()



	for option, value in optlist:

		if option=="--help":

			usage()



		elif option=="--httpproxy":

			proxies["http"]=value

		elif option=="--ftpproxy":

			proxies["ftp"]=value

		elif options=="--gopherproxy":

			proxies["gopher"]=value



	urls=args



	for url in urls:

		try:

			outfile=open(getURLName(url), "wb")

			fileName=outfile.name.split(os.sep)[-1]



			url, length=createDownload(url, proxies)

			if not length:

				length="?"



			print "Downloading %s (%s bytes) ..." % (url.url, length)

			if length!="?":

				length=float(length)

			bytesRead=0.0



			for line in url:

				bytesRead+=len(line)



				if length!="?":

					print "%s: %.02f/%.02f kb (%d%%)" % (

						fileName,

						bytesRead/1024.0,

						length/1024.0,

						100*bytesRead/length

					)



				outfile.write(line)



			url.close()

			outfile.close()

			print "Done"



		except Exception, e:

			print "Error downloading %s: %s" % (url, e)



if __name__=="__main__":

	main()

How to download files with Ruby

TDonaghe (Terry Donaghe) — Fri, 25 Aug 2006 19:59:34 GMT

// Note: the "b" in "wb" in the open method may not be needed in
// non-Windows environments. In Windows it indicates that you're writing
// binary information. You probably won't need it for downloading straight text
// or html either.



require 'net/http'



Net::HTTP.start("static.flickr.com") { |http|

  resp = http.get("/92/218926700_ecedc5fef7_o.jpg")

  open("fun.jpg", "wb") { |file|

    file.write(resp.body)

   }

}

puts "Yay!!"

Download recent flickr pictures with ruby and the flickr api

TDonaghe (Terry Donaghe) — Fri, 25 Aug 2006 19:51:25 GMT

// To make this work, you need to get your own flickr api key.
// Get one here: http://www.flickr.com/services/api/misc.api_keys.html
// Other than that, just plug and chug and have fun!
// The "b" in "wb" in the second open method may not be necessary in
// non-windows environments.



require 'open-uri'

require 'rexml/document'



open('http://www.flickr.com/services/rest/?method=flickr.photos.getRecent&api_key=YOUR_KEY_HERE') { |f|

    doc = REXML::Document.new f.read

    i = 0

    doc.elements.each("rsp/photos/photo") { |element|

        if i < 3

            open("images/file" << i.to_s << ".jpg", "wb").

                write(open("http://static.flickr.com/" << \

                element.attributes["server"] << "/" << \

                element.attributes["id"] << "_" << \

                element.attributes["secret"] << "_o.jpg").read)

        else

            break

        end

        i = i + 1

    }

}



puts "Done!"

Force output to download as a file in Rails

timmorgan (Tim Morgan) — Tue, 16 May 2006 23:32:04 GMT



response.headers['Content-Type'] = 'text/csv' # I've also seen this for CSV files: 'text/csv; charset=iso-8859-1; header=present'

response.headers['Content-Disposition'] = 'attachment; filename=thefile.csv'

Download the pys60 tutorial

korakot (Korakot Chaovavanich) — Wed, 01 Feb 2006 13:05:19 GMT

Many wonderful examples by Jürgen Scheible.
I omit the zip,exe,pdf to save site's bandwidth.



C:\>wget -kmnp -l3 -Rzip,exe,pdf http://www.mobilenin.com/pys60/menu.htm

You can just browse the site.

Mirror a site with wget

korakot (Korakot Chaovavanich) — Mon, 11 Apr 2005 05:07:04 GMT

I create a batch file (w.bat) containing just



wget -krmnp %1

Then I call



C:\>w [what-ever-url]