DZone Snippets: Joeldg's Code Snippets

weighted distribution functions

joeldg (joeldg) — Thu, 18 Jan 2007 00:03:27 GMT

I had to write a scheduler once upon a time and this was what I ended up using to do the schedule for users.. days had different weights, and certain users were management and others not (management didn't want to be scheduled on weekends.. go figure)..

works well, and I am pretty sure that five years later this function is still getting a lot of use.



# users "user"=>weight, "user"=>weight ...

$users = array("manager"=>1,"peon"=>4, "middlemngr"=>3, "sales"=>3, "other peon"=>4);

asort($users);



############################################################################################################

function distrib($arr){

    $weight_total = array_sum($arr);

    $vals = array_values($arr);

    

    // loop through and figure out the percentage for each user.

    while ( list($key, $val) = each($arr) ){

       $curr = $val;

       //$percent =  (($curr * 100) / $total); // percentage of highest..

       $percent =  (($curr / $weight_total) * 100); // this is for % of sum(total)

       $percent_int = floor($percent);

           $percent_float = number_format($percent, 1);    

       

       // now set up our return array.. "user"=>percentage, "user"=>percentage ...

       $ret[$key] = $percent_float;

    }//wend

  // send it back

  return $ret;

}// end function



############################################################################################################

// this function spits back a placement for a user..

// theoretically this should have an even distribution based on our distribution list.

// dist_arr = output from distrib()

function determine_placement($dist_arr, $rndval=""){

    if ($rndval == "") { $rndval = mt_rand(0,100); } //set a rnd val

    $running = 0; // private $running default

    while (list($key, $val) = each ($dist_arr)){

        $running  = $val;

        if ($rndval < $running) {

            return $key;

        }

    }//wend

    return false;

}//end function



############################################################################################################

// just to test the output

function test_output($dist_arr){

    while (list($key, $val) = each ($dist_arr)){

        echo "$key  ($val%)\n";

    }//wend

}//end function



$gen_distrib = distrib($users);

?>

useful normalizing functions (php)

joeldg (joeldg) — Wed, 17 Jan 2007 23:53:22 GMT

useful normalizing functions for transfer of data into a neural net safe array (0-1) or for fitting data for a graph etc..

can also be used in reverse by using the array largest and least values.



function normalize($arr,$LO=0.01,$HI=0.99)

{

  $Min =  2147483647;

  $Max = -2147483647;

  for ($a=0; $a    $Min = min($Min, $arr[$a]);

    $Max = max($Max, $arr[$a]);

  }

  $Mean = 0;

  for ($a=0; $a    $div = $Max-$Min;

    if($div == 0){$div = 1;}

    $arr [$a] = (($arr[$a]-$Min) / ($div)) * ($HI-$LO)   $LO;

    $Mean  = $arr[$a] / count($arr);

  }

  return $arr;

}





/*

    dual-way recursive greedy grabber..

    begins at a position in an array and grabs all "in order" parts up and

    down from that point and returns them preserving keys.

*/

function section_greedy($stack, $start=0, $stop=2){

    $check2 = array_slice_key ($stack, $start-1, $stop);

    if ($start >= 0 && $stop <= count($stack) 1){ // limit recursion to bounds.

    if (is_in_rorder_key($check2)){

        $check =  $check2;

        $begin = $start-1; $ending = $stop;        

        $check3 = section_greedy($start-1, $stop 1); //recurse up

        if(is_in_rorder_key($check3)){

            $check = $check3;

            $moremore = section_greedy($start, $stop 1); //recurse down

            if(is_in_rorder_key($moremore)){

                $check =  $moremore;

                $begin = $start-2; $ending = $stop 2;

            }//fi

        }//fi

    }else{

        $check =  array_slice_key($stack, $start, $stop);

        $begin = $start; $ending = $stop;

    }//fi

    }//fi

    return $check;

}//end function



/*

returns an array of movements up/down and the change.

Array(

    [down] => 1178.36

    [up] => 419.05

    [change] => -759.31

)

*/

function examine_movement($arr){

    while(list($key,$val) = each($arr)){

        if($oldval == ""){$oldval = $val;}

        if($val > $oldval){

            $ret[up] += $val-$oldval;

        }else{

            $ret[down] += $oldval-$val;

        }

        $oldval = $val;        

    }//wend

    $ret[change] = $ret[up] - $ret[down];

    return $ret;

}



?>

Colorized text from php in unix terminals

joeldg (joeldg) — Tue, 16 Jan 2007 23:30:29 GMT

Colorized text from php in unix terminals



# first define colors to use

$_colors = array(

        LIGHT_RED      => "[1;31m",

        LIGHT_GREEN     => "[1;32m",

        YELLOW         => "[1;33m",

        LIGHT_BLUE     => "[1;34m",

        MAGENTA     => "[1;35m",

        LIGHT_CYAN     => "[1;36m",

        WHITE         => "[1;37m",

        NORMAL         => "[0m",

        BLACK         => "[0;30m",

        RED         => "[0;31m",

        GREEN         => "[0;32m",

        BROWN         => "[0;33m",

        BLUE         => "[0;34m",

        CYAN         => "[0;36m",

        BOLD         => "[1m",

        UNDERSCORE     => "[4m",

        REVERSE     => "[7m",



);

##############################################

# Output colorized text to terminal run

# php scripts..

##############################################

function termcolored($text, $color="NORMAL", $back=1){

    global $_colors;

    $out = $_colors["$color"];

    if($out == ""){ $out = "[0m"; }

    if($back){

        return chr(27)."$out$text".chr(27)."[0m";#.chr(27);

    }else{

        echo chr(27)."$out$text".chr(27).chr(27)."[0m";#.chr(27);

    }//fi

}// end function

##############################################

terminal live text graph in php, unix terminal

joeldg (joeldg) — Tue, 16 Jan 2007 23:11:44 GMT

I often run php scripts from the command line in linux and need a way to view time series of data, often I need to see the number of mysql queries or want to view other stats.
That look like the following, it moves from right to left in real time as the data points change.



+local mysql: 565 queries/sec 189----------------------+

 |                                                      |

 |                                       %              |

 |                                       %              |

 |                                       %              |

 |                                       %              |

 |                                       %              |

 |                   %                   %              |

 | %   %%   %% % %  %%%%%                % %%        %% |

 |%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%% %%%%%%  %%%%%|

 |%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%|

+-------------------------------------high: 100, low: 1+

Just copy and past the following code into a file called "termgraph.php" and change the mysql password (if needed) and change the database name (if needed as it it set to work with the default mysql install), then run it and see, very easy to modify this to work with whatever you need. In addition, you can also do colored graphs in terminals using another function I am going to post here.

This needs a couple changes to work with your mysql database, but the graphs are fun to watch, hope you enjoy this.



// see *EDIT* below for one other change you will need to make

function DatabaseConnect() {

    if (!($mylink = mysql_connect("localhost", "root", ""))){

            print  "ERROR";

            exit;

        }//fi

        mysql_select_db("test") or die(mysql_error());

}// end function

DatabaseConnect();



/*

    remove flatspots in array..

    areas where the data does not change.

    this is good for data that only changes during certain times

    and the dead time has no bearing on the changes.

    -joeldg

*/

function remove_flatspots($arr, $pkey=true){

    while(list($key,$val) = each($arr)){

        if($val <> $oldval){

            if($pkey == true){

                $ret[$key] = $val;

            }else{

                $ret[] = $val;

            }

        }

        $oldval = $val;

    }

    return $ret;

}

/*

    take two arrays, remove the flatspots in the first.

    return an array containing the first with it's corresponding

    values in the second array..

    -joeldg

*/

function dual_remove_flatspots($arr1, $arr2, $pkey=true){

    while(list($key,$val) = each($arr1)){

        if($val <> $oldval){

            if($pkey == true){

                $ret[0][$key] = $val;

                $ret[1][$key] = $arr2[$key];

            }else{

                $ret[0][] = $val;

                $ret[1][] = $arr2[$key];

            }

        }

        $oldval = $val;

    }

    return $ret;

}



// return lowest val of array -joeldg

function least($inarr){

    $ret = $inarr[0];

    for($i=0;$i        if(intval($inarr[$i]) <= $ret){ $ret = $inarr[$i]; }

    }//rof

    return $ret;

}// end function

// return higest val of array -joeldg

function most($inarr){

    while(list($key,$val) = each($inarr)){

        if ($ret==""){$ret = $val;}

        if($ret <= intval($val)){ $ret = $val; }

    }//rof

    return $ret;

}// end function

/*

	array normalize function 

	-joeldg

*/

function normalize($arr,$LO=0.01,$HI=0.99)

{

  $Min = +2147483647;

  $Max = -2147483647;

  for ($a=0; $a    $Min = min($Min, $arr[$a]);

    $Max = max($Max, $arr[$a]);

  }

  $Mean = 0;

  for ($a=0; $a    $div = $Max-$Min;

    if($div == 0){$div = 1;}

    $arr [$a] = (($arr[$a]-$Min) / ($div)) * ($HI-$LO) + $LO;

    $Mean += $arr[$a] / count($arr);

  }

  return $arr;

}

/*

	array normalize function, preserve key

	-joeldg

*/

function normalizekey($arr,$LO=0.01,$HI=0.99)

{

  $Min = +2147483647;

  $Max = -2147483647;

  while(list($key, $a)=each($arr)){

    $Min = min($Min, $arr[$key]);

    $Max = max($Max, $arr[$key]);

  }

  reset($arr);

  $Mean = 0;

  while(list($key,$a)=each($arr)){

    $div = $Max-$Min;

    if($div == 0){$div = 1;}

    $arr [$key] = (($arr[$key]-$Min) / ($div)) * ($HI-$LO) + $LO;

    $Mean += $arr[$key] / count($arr);

  }

  $retarr[0] = $arr;

  $retarr[1][0]=$Max;

  $retarr[1][1]=$Min;

  

  return $retarr;

}



/*

display a textual graph in an xterm.

written because I want to view timeseries data and not have to jump over to a browser.

*/

#

#    transform an array point to a position within the total

#

function point2arr($point, $total=20){

    $p = round($point);

    for($a=0;$a<$total;$a++){

        if($a <= $p){

            $ret[] = "%";#chr(127);

        }else{

            $ret[] = " ";

        }

    }

    return $ret;

}

// get the width and height of a unix terminal

function get_term_specs(){

    $b = `stty -a`;

    $c = explode("\n",$b);

    $d = explode(";", $c[0]);

    $f = explode(" ",$d[1]);

    $ret[h] = $f[2];

    $f = explode(" ",$d[2]);

    $ret[w] = intval($f[2]);    

    return $ret;

}



function genchars($char, $total, $title="", $echo=false, $way=STR_PAD_RIGHT){

    $ret .= "+";

    #if($echo){ echo termcolored("+", WHITE); }

    #if(!$echo){ $ret .= "+"; }

        #$back = termcolored($title, "YELLOW");

        $back = $title;

        $ret .= str_pad($back, $total+strlen($back)-strlen($title)-2, $char, $way);

    #if(!$echo){ $ret .= "+"; }

    #if($echo){ echo termcolored("+", WHITE); }

    $ret .= "+";

    return $ret;

}

// vertical graph

function print_vert_graph($arr, $total=20, $border="-"){

    $arr = normalize($arr, 0, $total);

    $out[] = genchars($border, $total);

    for($a=0;$a        $out[] = point2arr($arr[$a], $total);

    }

    $out[] = genchars($border, $total);

    for($a=0;$a        for($b=0;$b            echo $out[$a][$b];

        }

        echo "\n";

    }

    

}

// transform matrix 

function transformmat($arr,$total=20){

    $width = count($arr[0]);

    $c=$width-1;

    for($w=0;$w<$width;$w++){

        for($a=0;$a            $ret[$c][$a] = $arr[$a][$w];

        }

        $c--;

    }

    return $ret;

}

// horizontal graphing

function print_horz_graph($arr, $total=20, $border="-", $title="", $w=""){

    if($w <> ""){

        array_reverse($arr);

        $end = count($arr)-1;

        reset($arr);

        while(list($key,$val)=each($arr)){

            $newarr[] = $arr[$key];

            $end--;

            if($end <= 0){break;}

        }

        #array_reverse($newarr);

        $arr=$newarr;

    }

    #print_R($arr);

    $bottom = "high: ".most($arr).", low: ".least($arr);

    $arr = normalize($arr, 0, $total-2);

    #$out[] = genchars($border, $total);

    for($a=0;$a        $out[] = point2arr($arr[$a], $total);

    }

    $out = transformmat($out,$total);

    $ret .= genchars("-", count($out[0])+2, $title, false);

    #echo "\n";

    $ret .= "\n";

    for($a=0;$a        #echo termcolored("|", WHITE);

        $ret .=  " |";

        for($b=0;$b            $ret .= $out[$a][$b];

        }

        $ret .= "|";

        #echo termcolored("|", WHITE);

        $ret .= "\n";

    }

    $ret .= genchars("-", count($out[0])+2, $bottom, false, STR_PAD_LEFT);

    $ret .= "\n";

    return $ret;

    

}



// *EDIT* change mysql to whatever table you want to test

// this default should work for fun..

$db_list = mysql_list_tables("mysql");

while ($row = mysql_fetch_row($db_list)) {

   $tables[] = trim("{$row[0]}");

}

$testarr_z = array();

$testarr = array();

$qcount = 0;

$start = microtime(true);



while(1){

	$hw = get_term_specs();

	array_pad($testarr_z, ($hw['w'] - 6), 0);



	array_pad($testarr, ($hw['w'] - 6), 0);

	$time_start = microtime(true);

	$tab = $tables[rand(0,count($tables))];

	

	$amt = rand(500, 5000);

	$sql = "SELECT * FROM $tab LIMIT 0,$amt";

	$res = mysql_db_query("oracle", $sql);

	$qcount++;

	

	$time_end = microtime(true);

	$time = $time_end - $time_start;

	$testarr_z[] = $time;

	@mysql_free_result($res);

	

	if(count($testarr_z)> ($hw['w'] - 6)){

		array_shift($testarr_z);

	}

	$testarr = normalize($testarr_z, 1, 100);



	$testarr = remove_flatspots($testarr, false);

	$tottime = $time_end - $start;

	$qps = round($qcount/$tottime);

	echo print_horz_graph($testarr, $hw['h']-3, "-", "local mysql: $qcount queries/sec $qps", $hw['w']-1);



}



?>

Rails - ‘poor mans’ SQL cache.

joeldg (joeldg) — Mon, 15 Jan 2007 22:55:31 GMT

Rails memcached is not very easy to introduce to a large rails installation. Memcached also chews up a lot of memory on the box and overall cached model does not work the way I needed it to. Basically, I have just a â€œfewâ€? queries that I needed to cache because pagination sucks just that bad in rails.
So, I built my own cache, similar to how I build them in PHP except I am not using disk cache, I am using MySQL itself to cache itâ€™s own results

First, we need a table to hold all this info (note the â€˜blobâ€™ field)



CREATE TABLE `cacheditems` (

`id` int(11) NOT NULL auto_increment,

`cachekey` varchar(255) default NULL,

`created` datetime default NULL,

`expires` datetime default NULL,

`content` longblob,

`cachehit` int(11) NOT NULL,

PRIMARY KEY  (`id`),

KEY `cacheditems_cachekey_index` (`cachekey`),

KEY `cacheditems_created_index` (`created`),

KEY `cacheditems_expires_index` (`expires`)

)

Then we create a model called â€œcacheditemâ€? which has the following functions



require 'digest/sha1'

class Cacheditem < ActiveRecord::Base



def self.checkfor(sql)

key = Digest::MD5.hexdigest(Marshal.dump(sql))

logger.info "%%% checking for key #{key}"

#logger.info "%%% checking by sql #{sql[0]}"

Cacheditem.find( :first, :conditions => [ â€œcachekey = ? AND expires > NOW()â€?, key] )

end



def self.getcached(sql)

key = Digest::MD5.hexdigest(Marshal.dump(sql))

logger.info â€œ%%% getting by key #{key}â€?

#logger.info â€œ%%% getting by sql #{sql[0]}â€?

getc = Cacheditem.find( :first, :conditions => [ â€œcachekey = ?â€?, key] )

hitcount = getc.cachehit + 1

Cacheditem.update(getc.id, {:cachehit => hitcount})

Cacheditem.delete_all â€œexpires < NOW()"  # cleaner

return Marshal.load( getc.content )

end



def self.storeresult(sql, result)

key = Digest::MD5.hexdigest(Marshal.dump(sql))

logger.info "%%% storing by key #{key}"

content = Marshal.dump(result)

logger.level = (4) # this stops display in logs of the marshal data

ci = new()

ci.cachekey    = key

ci.created       = Time.now()

ci.expires        = 30.minutes.from_now() # change as needed

ci.content        = content

ci.cachehit       = 0

ci.save

return  result

end



end

Then, in application.rb I added the following function



def find_by_sql_cache(sql)

iscached = Cacheditem.checkfor(sql)

if iscached

Cacheditem.getcached(sql)

else

result = connection.select_all(sanitize_sql(sql), "#{name} Load").collect! { |record| instantiate(record) }

Cacheditem.storeresult(sql, result)

end

end

just throw â€œ_cacheâ€? after any â€œfind_by_sqlâ€? statement you have a need to cache and there you are.

This works very fast, very well, and doesnâ€™t hog your memory. It cleans up after itself in the database, and perhaps it does that too much.. It would be easy to add in a standard garbage collection function which runs on a random but I felt this gave me much better stats of the actual thirty-minute cacheâ€¦
If you use zabbix for monitoring your network, you can have fun graphs of cache statistics by adding the following to your zabbix_agentd.conf



UserParameter=mysql.totalcache,mysql â€“batch â€“skip-column-names -D {YOUR_DATABASE} -u{YOUR_USERNAME} -p{YOUR_PASSWORD} -e â€œSELECT count( * ) AS total, SUM( cachehit ) AS amount from cacheditems;â€? | cut -f1

UserParameter=mysql.cachehits,mysql â€“batch â€“skip-column-names -D {YOUR_DATABASE} -u{YOUR_USERNAME} -p{YOUR_PASSWORD} -e â€œSELECT count( * ) AS total, SUM( cachehit ) AS amount from cacheditems;â€? | cut -f2

DZone Snippets: Joeldg's Code Snippets

weighted distribution functions

useful normalizing functions (php)

Colorized text from php in unix terminals

terminal *live* text graph in php, unix terminal

Rails - ‘poor mans’ SQL cache.

terminal live text graph in php, unix terminal