I'm still working on getting this one perfect.

chmod 701 /home/*
chmod 705 /home/*/public_html
chmod 604 /home/*/public_html/*.*

Replace aaa.bbb.ccc.ddd with your own IP (so you always get to your site).
Replace www.yourdomain.com with your actual domain.

Just don’t mess with the user_agent and query_string lines. Those ensure that the Coral servers themselves can retrieve your page when they need to.

(Shamelessly stolen from http://ottodestruct.com/diggprotectionrules.txt)

RewriteEngine on
RewriteCond %{REMOTE_ADDR} !^aaa.bbb.ccc.ddd$
RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$
RewriteCond %{HTTP_REFERER} ^http://(www\.)?digg\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?slashdot\.org [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?slashdot\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?fark\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?somethingawful\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?kuro5hin\.org [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?engadget\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?boingboing\.net [OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?del\.icio\.us
RewriteRule ^(.*)$ http://www.yourdomain.com.nyud.net:8080$1 [R,L]

The following code is the contents of /banme/index.php. This file is linked to from my main website but invisible to web browsers and disallowed in robots.txt. Therefore, only bad bots will ever follow this link and when they do so they will get banned in .htaccess and their ip address will be emailed to webmaster@example.com.

<?php
$i = getenv('REMOTE_ADDR');
$handle = fopen("../.htaccess", "a");
fwrite($handle, "Deny from $i\n");
fclose($handle);
echo "You've just got $i banned from this domain.  You are a very bad person.";
mail("webmaster@example.com", "Banned IP", "Deny from $i");
?>

// description of your code here

<?php
/*
Plugin Name: Googleit
Plugin URI: http://lordrich.com/archives/2005/04/02/just-google-it/
Description: Link to google for the current title.  Usage: google_it();
Version: 0.1
Author: Richard Kirkcaldy
Author URI: http://lordrich.com
*/

function google_it(){
	$google = '<a href="http://www.google.com/search?q='.get_the_title().'">Google It</a>';
	echo $google;
	}
?>

Display a list of email addresses for people who have filed tickets in the past 30 days.

<?php
$database = 'cerberus';
$username = 'cerberus';
$password = 'cerberus';
$hostname = 'localhost';

$sql = "
SELECT distinct address.address_address
FROM ticket, requestor, address
WHERE  DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= last_update_date
AND ticket_status='resolved'
AND ticket.ticket_id=requestor.ticket_id
AND requestor.suppress='0'
AND requestor.address_id=address.address_id";

mysql_connect($hostname,$username,$password) or die('error connecting to the database');
@mysql_select_db($database) or die('unable to select database');
$result = mysql_query($sql) or die(mysql_error());
$num=mysql_numrows($result);
mysql_close();

$i = 0;
while($i < $num) {
	$address_address = mysql_result($result,$i,"address.address_address");
	echo "$address_address<br/>";
	$i++;
}
?>

// description of your code here

<?php

    // Displays a single random photo from recent flickr photos with a given tag.
    // Original code stolen from many sources including http://www.thebishop.net/geodog/archives/2004/09/29/fun_hacking_with_flickr_making_a_homemade_flickr_tag_badge_with_magpierss.html and http://prwdot.org/archives/002468.html


    // USER CONFIGURATION SECTION

    // MagpieRSS Configuration
    // This is an example based on my system;
    // you will need to customize it for your system
    // and your preferences. You can remove it entirely
    // if you have done it elsewhere
    // refer to http://magpierss.sourceforge.net/
    require_once('/var/www/bradford/magpierss-0.61/rss_fetch.inc');
    error_reporting(E_ERROR);
    define(MAGPIE_CACHE_ON, true);
    define(MAGPIE_CACHE_DIR, '/var/www/bradford/magpie_cache');
    define(MAGPIE_CACHE_AGE, 300);
    define(MAGPIE_CACHE_FRESH_ONLY, false);
    define(MAGPIE_DETECT_ENCODING, true);
    define(MAGPIE_DEBUG, 0);
    define(MAGPIE_FETCH_TIME_OUT, 15);
    define(MAGPIE_USE_GZIP, true);    

    // flickr configuration
    // How many photos you want to display
    $num_photos = 2; // for some reason it doesn't like 1
    $tag = 'Bradford';
    // URL for the flickr feed you want to use
    $flickr_feed_url ='http://www.flickr.com/services/feeds/photos_public.gne?tags='.$tag.'&format=rss_200';
?>

<?php
    // Fetch the feed
    $flickr = fetch_rss( $flickr_feed_url );
    if ($flickr) {
        $flickr_title = $flickr->channel["title"];
        $flickr_link = $flickr->channel["link"];
?>

<!-- Display the title and link to the feed -->

<?php
    // Pick some random photos
        $random_photos = array_rand($flickr->items,$num_photos);
        foreach ( $random_photos as $random_photo ) {
            $description = explode("\n\n",$flickr->items[$random_photo]["description"]);
?>

    <!-- Display the given photo -->

	<?php echo ereg_replace('<img src=(.*) width=(.*)>', '<img src=\\1 width="150px"/>', $description[1]);
	  die();?> // ok we've got our first photo - lets exit

<?php
        }
      } else {
?>

<!-- Display an error message if things didn't work -->
<p>An error occurred in the MagpieRSS parser:</p>

<p><?php echo magpie_error(); ?></p>

<?php
    } 
?>

An SMTP client in CPP, apparently for a Linux machine. It was found in my old source code folder and so may not be fully working.

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdio.h>

#define HELO "HELO\n"
#define DATA "DATA\n"
#define QUIT "QUIT\n"

FILE *fin;
int sock;
struct sockaddr_in server;
struct hostent *hp, *gethostbyname();
char buf[BUFSIZ+1];
int len;
char *host_id;
char *from_id;
char *to_id;
char *file_id;
char wkstr[100];

/*=====Send a string to the socket=====*/

send_socket(char *s)
{
	write(sock,s,strlen(s));
	write(1,s,strlen(s));
}

/*=====Read a string from the socket=====*/

read_socket()
{
	len = read(sock,buf,BUFSIZ);
	write(1,buf,len);
}

/*=====MAIN=====*/
int main(int argc, char* argv[])
{

if(argc != 5)
{
 printf("USAGE: %s <host> <from> <to> <filename>\n\n", argv[0]);
 exit(1);
}

host_id=argv[1];
from_id=argv[2];
to_id=argv[3];
file_id=argv[4];

/*=====Create Socket=====*/
sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock==-1)
{
 perror("opening stream socket");
 exit(1);
}

/*=====Verify host=====*/
server.sin_family = AF_INET;
hp = gethostbyname(host_id);
if (hp==(struct hostent *) 0)
{
 fprintf(stderr, "%s: unknown host\n", host_id);
 exit(2);
}

/*=====Connect to port 25 on remote host=====*/
printf ("hostent %s\n", hp->h_addr_list[0]);
memcpy((char *) &server.sin_addr, (char *) hp->h_addr, hp->h_length);

server.sin_port=htons(25); /* SMTP PORT */

if (connect(sock, (struct sockaddr *) &server, sizeof server)==-1)
{
 perror("connecting stream socket");
 exit(1);
}

/*=====Write some data then read some =====*/

read_socket(); /* SMTP Server logon string */

send_socket(HELO); /* introduce ourselves */
read_socket(); /*Read reply */

send_socket("MAIL from: "); /* Mail from us */
send_socket(from_id);
send_socket("\n");
read_socket(); /* Sender OK */

send_socket("RCPT To: "); /*Mail to*/
send_socket(to_id);
send_socket("\n");
read_socket(); /*Recipient OK*/

send_socket(DATA);/*body to follow*/
read_socket(); /*ok to send */

fin=fopen(file_id, "r"); /* open file */
while(1)
{
 if(fgets(wkstr, 100, fin)==NULL) break; /* exit on EOF */
 send_socket(wkstr);
}
fclose(fin); /* close file */

send_socket(fin); /*send file*/
send_socket(".\n");

read_socket(); /* OK*/
send_socket(QUIT); /* quit */
read_socket(); /* log off */

/*=====Close socket and finish=====*/
close(sock);
exit(0);
}

Again found in my old source folder, it may not fully work.

This Perl script reads in the existing links from links.dat into the array @bigarray. It then loops through the array reading in each link and appending the new links it finds to links.dat. If the script were run in a loop it would add every single web address it can find to links.dat.

#!/usr/bin/perl 
use IO::Socket; 
use URI; 
 
open(LINKS, "<< links.dat"); 
@bigarray = (); 
while (<LINKS>) { 
        chomp; 
        push(@bigarray, $_); 
} 
close(LINKS); 
 
foreach $uri (@bigarray) { 
        ($domain = URI->new($uri)->authority) =~ s/^www\.//i; 
        $socket = IO::Socket::INET->new(PeerAddr 
                                => $domain, 
                                PeerPort => 80, 
                                Proto => 'tcp', 
                                Type => SOCK_STREAM) 
        or die "Couldn't connect"; 
        print $socket "GET / HTTP/1.0\n\n"; 
        #$page = <$socket>; 
        open(LINKS, ">> links.dat"); 
        while (defined($line = <$socket>)) { 
                $line =~ m{href="(.*?)"}ig; 
                print LINKS "$1"; 
            } 
        close(LINKS); 
        close($socket); 
}

Example output:

-bash-2.05b$ php asp.php
http://www.example.com
http://www.rfc-editor.org/rfc/rfc2606.txt
No links.

-bash-2.05b$ cat links.dat
http://www.example.com
http://www.rfc-editor.org/rfc/rfc2606.txt

<?php
$datafile = "links.dat"; // file to keep the list of links in
$regex = "/<\s*a\s+[^>]*href\s*=\s*[\"']?([^\"' >]+)[\"' >]/isU";  // regex to search for hrefs

$handle = fopen($datafile, "r"); // open the data file
$buffer = fgets($handle, 4096);
$oldlinks[] = $buffer; // read the first link into an array
while (!feof($handle)) {
	$buffer = fgets($handle, 4096);
	array_push($oldlinks,$buffer); // read the rest of the links into an array
}
fclose($handle); // close the data file

foreach($oldlinks as $value) { // for every link in the array
	print $value; // print it out
	$remote = fopen(trim($value), "r") or die(); //open it or fail nicely
	while (!feof($remote)) {
		$html = fread($remote, 8192); // read in the remote page
	}
	fclose($remote); // close it
	if (preg_match_all($regex, $html, $links)) { // if we find new links
		$local = fopen($datafile, "a+"); // open the data file
		foreach($links[1] as $value) { // for every new link
			$value.="\n"; // append a new line
			if(!in_array($value,$oldlinks)) { // if we haven't seen it before (nb - case sensitive)
				print($value); // print it out
				fwrite($local, $value); // and write it to file
			}
		}
		fclose($local); // close the data file
	}
	else {
		print("No links."); // we didn't find any links in the new file
	}
}
?>

NB. Again, this script was found in my old source code folder, it may not be fully working.

This Ruby script reads in a list of links from links.dat, it then picks out the ones it can easily spider and gets a list of URLs from each page listed in links.dat. Every new URL it finds will be added to newlinks.dat for later spidering by another bot running along side this one.

require 'socket' 
links = File.open("links.dat") 
while links.gets do 
        #domain = ($_ =~ /http:\/\/.*\.([0-9a-zA-Z\-]+\.com|net|org)/); 
        if %r{http://([^/]+)/([^/]+)}i =~ $_ 
                domain,path = $1, $2 
        end 
        if proto="http" 
                begin 
                        t = TCPSocket.new(domain, 'www') 
                rescue 
                        puts "error: #{$!}" 
                else 
                        t.print "GET /"+path+" HTTP/1.0\n\n" 
                        answer = t.gets(nil) 
                        t.close 
                end 
 
                if %r{<a\s+href="(\w+)://([^"]+)"[^>]*>([^<]*)</a>}i =~ answer 
                        proto, url, text = $1, $2, $3 
                end 
 
                print proto+"://"+url+"\n" 
                old = File.open("newlinks.dat") 
                new = File.open("links.dat.tmp", File::WRONLY|File::TRUNC|File::CREAT) 
                while old.gets do 
                        if $_ != proto+"://"+url 
                                new.print $_ 
                        end 
                end 
                new.print proto+"://"+url 
                old.close 
                new.close 
                File.rename("newlinks.dat", "links.dat.orig") 
                File.rename("links.dat.tmp", "newlinks.dat") 
        end 
end 
links.close