Never been to DZone Snippets before?

Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

About this user

joeldg http://blog.peoplesdns.com

« Newer Snippets
Older Snippets »
Showing 1-5 of 5 total  RSS 

weighted distribution functions

I had to write a scheduler once upon a time and this was what I ended up using to do the schedule for users.. days had different weights, and certain users were management and others not (management didn't want to be scheduled on weekends.. go figure)..

works well, and I am pretty sure that five years later this function is still getting a lot of use.

<?
# users "user"=>weight, "user"=>weight ...
$users = array("manager"=>1,"peon"=>4, "middlemngr"=>3, "sales"=>3, "other peon"=>4);
asort($users);

############################################################################################################
function distrib($arr){
    $weight_total = array_sum($arr);
    $vals = array_values($arr);
    
    // loop through and figure out the percentage for each user.
    while ( list($key, $val) = each($arr) ){
       $curr = $val;
       //$percent =  (($curr * 100) / $total); // percentage of highest..
       $percent =  (($curr / $weight_total) * 100); // this is for % of sum(total)
       $percent_int = floor($percent);
           $percent_float = number_format($percent, 1);    
       
       // now set up our return array.. "user"=>percentage, "user"=>percentage ...
       $ret[$key] = $percent_float;
    }//wend
  // send it back
  return $ret;
}// end function

############################################################################################################
// this function spits back a placement for a user..
// theoretically this should have an even distribution based on our distribution list.
// dist_arr = output from distrib()
function determine_placement($dist_arr, $rndval=""){
    if ($rndval == "") { $rndval = mt_rand(0,100); } //set a rnd val
    $running = 0; // private $running default
    while (list($key, $val) = each ($dist_arr)){
        $running  = $val;
        if ($rndval < $running) {
            return $key;
        }
    }//wend
    return false;
}//end function

############################################################################################################
// just to test the output
function test_output($dist_arr){
    while (list($key, $val) = each ($dist_arr)){
        echo "<td width=\"$val%\">$key  ($val%)</td>\n";
    }//wend
}//end function

$gen_distrib = distrib($users);
?>

<table width=1 border=1 cellpadding=0 cellspacing=0>
<tr>
<? test_output(distrib($placement)); ?>
</tr>
</table>

useful normalizing functions (php)

useful normalizing functions for transfer of data into a neural net safe array (0-1) or for fitting data for a graph etc..

can also be used in reverse by using the array largest and least values.

<?
function normalize($arr,$LO=0.01,$HI=0.99)
{
  $Min =  2147483647;
  $Max = -2147483647;
  for ($a=0; $a<count($arr); $a++) {
    $Min = min($Min, $arr[$a]);
    $Max = max($Max, $arr[$a]);
  }
  $Mean = 0;
  for ($a=0; $a<count($arr); $a++) {
    $div = $Max-$Min;
    if($div == 0){$div = 1;}
    $arr [$a] = (($arr[$a]-$Min) / ($div)) * ($HI-$LO)   $LO;
    $Mean  = $arr[$a] / count($arr);
  }
  return $arr;
}


/*
    dual-way recursive greedy grabber..
    begins at a position in an array and grabs all "in order" parts up and
    down from that point and returns them preserving keys.
*/
function section_greedy($stack, $start=0, $stop=2){
    $check2 = array_slice_key ($stack, $start-1, $stop);
    if ($start >= 0 && $stop <= count($stack) 1){ // limit recursion to bounds.
    if (is_in_rorder_key($check2)){
        $check =  $check2;
        $begin = $start-1; $ending = $stop;        
        $check3 = section_greedy($start-1, $stop 1); //recurse up
        if(is_in_rorder_key($check3)){
            $check = $check3;
            $moremore = section_greedy($start, $stop 1); //recurse down
            if(is_in_rorder_key($moremore)){
                $check =  $moremore;
                $begin = $start-2; $ending = $stop 2;
            }//fi
        }//fi
    }else{
        $check =  array_slice_key($stack, $start, $stop);
        $begin = $start; $ending = $stop;
    }//fi
    }//fi
    return $check;
}//end function

/*
returns an array of movements up/down and the change.
Array(
    [down] => 1178.36
    [up] => 419.05
    [change] => -759.31
)
*/
function examine_movement($arr){
    while(list($key,$val) = each($arr)){
        if($oldval == ""){$oldval = $val;}
        if($val > $oldval){
            $ret[up] += $val-$oldval;
        }else{
            $ret[down] += $oldval-$val;
        }
        $oldval = $val;        
    }//wend
    $ret[change] = $ret[up] - $ret[down];
    return $ret;
}

?>

Colorized text from php in unix terminals

Colorized text from php in unix terminals

# first define colors to use
$_colors = array(
        LIGHT_RED      => "[1;31m",
        LIGHT_GREEN     => "[1;32m",
        YELLOW         => "[1;33m",
        LIGHT_BLUE     => "[1;34m",
        MAGENTA     => "[1;35m",
        LIGHT_CYAN     => "[1;36m",
        WHITE         => "[1;37m",
        NORMAL         => "[0m",
        BLACK         => "[0;30m",
        RED         => "[0;31m",
        GREEN         => "[0;32m",
        BROWN         => "[0;33m",
        BLUE         => "[0;34m",
        CYAN         => "[0;36m",
        BOLD         => "[1m",
        UNDERSCORE     => "[4m",
        REVERSE     => "[7m",

);
##############################################
# Output colorized text to terminal run
# php scripts..
##############################################
function termcolored($text, $color="NORMAL", $back=1){
    global $_colors;
    $out = $_colors["$color"];
    if($out == ""){ $out = "[0m"; }
    if($back){
        return chr(27)."$out$text".chr(27)."[0m";#.chr(27);
    }else{
        echo chr(27)."$out$text".chr(27).chr(27)."[0m";#.chr(27);
    }//fi
}// end function
##############################################

terminal *live* text graph in php, unix terminal

I often run php scripts from the command line in linux and need a way to view time series of data, often I need to see the number of mysql queries or want to view other stats.
That look like the following, it moves from right to left in real time as the data points change.
+local mysql: 565 queries/sec 189----------------------+
 |                                                      |
 |                                       %              |
 |                                       %              |
 |                                       %              |
 |                                       %              |
 |                                       %              |
 |                   %                   %              |
 | %   %%   %% % %  %%%%%                % %%        %% |
 |%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%% %%%%%%  %%%%%|
 |%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%|
+-------------------------------------high: 100, low: 1+

Just copy and past the following code into a file called "termgraph.php" and change the mysql password (if needed) and change the database name (if needed as it it set to work with the default mysql install), then run it and see, very easy to modify this to work with whatever you need. In addition, you can also do colored graphs in terminals using another function I am going to post here.

This needs a couple changes to work with your mysql database, but the graphs are fun to watch, hope you enjoy this.

<?
// see *EDIT* below for one other change you will need to make
function DatabaseConnect() {
    if (!($mylink = mysql_connect("localhost", "root", ""))){
            print  "ERROR";
            exit;
        }//fi
        mysql_select_db("test") or die(mysql_error());
}// end function
DatabaseConnect();

/*
    remove flatspots in array..
    areas where the data does not change.
    this is good for data that only changes during certain times
    and the dead time has no bearing on the changes.
    -joeldg
*/
function remove_flatspots($arr, $pkey=true){
    while(list($key,$val) = each($arr)){
        if($val <> $oldval){
            if($pkey == true){
                $ret[$key] = $val;
            }else{
                $ret[] = $val;
            }
        }
        $oldval = $val;
    }
    return $ret;
}
/*
    take two arrays, remove the flatspots in the first.
    return an array containing the first with it's corresponding
    values in the second array..
    -joeldg
*/
function dual_remove_flatspots($arr1, $arr2, $pkey=true){
    while(list($key,$val) = each($arr1)){
        if($val <> $oldval){
            if($pkey == true){
                $ret[0][$key] = $val;
                $ret[1][$key] = $arr2[$key];
            }else{
                $ret[0][] = $val;
                $ret[1][] = $arr2[$key];
            }
        }
        $oldval = $val;
    }
    return $ret;
}

// return lowest val of array -joeldg
function least($inarr){
    $ret = $inarr[0];
    for($i=0;$i<count($inarr);$i++){
        if(intval($inarr[$i]) <= $ret){ $ret = $inarr[$i]; }
    }//rof
    return $ret;
}// end function
// return higest val of array -joeldg
function most($inarr){
    while(list($key,$val) = each($inarr)){
        if ($ret==""){$ret = $val;}
        if($ret <= intval($val)){ $ret = $val; }
    }//rof
    return $ret;
}// end function
/*
	array normalize function 
	-joeldg
*/
function normalize($arr,$LO=0.01,$HI=0.99)
{
  $Min = +2147483647;
  $Max = -2147483647;
  for ($a=0; $a<count($arr); $a++) {
    $Min = min($Min, $arr[$a]);
    $Max = max($Max, $arr[$a]);
  }
  $Mean = 0;
  for ($a=0; $a<count($arr); $a++) {
    $div = $Max-$Min;
    if($div == 0){$div = 1;}
    $arr [$a] = (($arr[$a]-$Min) / ($div)) * ($HI-$LO) + $LO;
    $Mean += $arr[$a] / count($arr);
  }
  return $arr;
}
/*
	array normalize function, preserve key
	-joeldg
*/
function normalizekey($arr,$LO=0.01,$HI=0.99)
{
  $Min = +2147483647;
  $Max = -2147483647;
  while(list($key, $a)=each($arr)){
    $Min = min($Min, $arr[$key]);
    $Max = max($Max, $arr[$key]);
  }
  reset($arr);
  $Mean = 0;
  while(list($key,$a)=each($arr)){
    $div = $Max-$Min;
    if($div == 0){$div = 1;}
    $arr [$key] = (($arr[$key]-$Min) / ($div)) * ($HI-$LO) + $LO;
    $Mean += $arr[$key] / count($arr);
  }
  $retarr[0] = $arr;
  $retarr[1][0]=$Max;
  $retarr[1][1]=$Min;
  
  return $retarr;
}

/*
display a textual graph in an xterm.
written because I want to view timeseries data and not have to jump over to a browser.
*/
#
#    transform an array point to a position within the total
#
function point2arr($point, $total=20){
    $p = round($point);
    for($a=0;$a<$total;$a++){
        if($a <= $p){
            $ret[] = "%";#chr(127);
        }else{
            $ret[] = " ";
        }
    }
    return $ret;
}
// get the width and height of a unix terminal
function get_term_specs(){
    $b = `stty -a`;
    $c = explode("\n",$b);
    $d = explode(";", $c[0]);
    $f = explode(" ",$d[1]);
    $ret[h] = $f[2];
    $f = explode(" ",$d[2]);
    $ret[w] = intval($f[2]);    
    return $ret;
}

function genchars($char, $total, $title="", $echo=false, $way=STR_PAD_RIGHT){
    $ret .= "+";
    #if($echo){ echo termcolored("+", WHITE); }
    #if(!$echo){ $ret .= "+"; }
        #$back = termcolored($title, "YELLOW");
        $back = $title;
        $ret .= str_pad($back, $total+strlen($back)-strlen($title)-2, $char, $way);
    #if(!$echo){ $ret .= "+"; }
    #if($echo){ echo termcolored("+", WHITE); }
    $ret .= "+";
    return $ret;
}
// vertical graph
function print_vert_graph($arr, $total=20, $border="-"){
    $arr = normalize($arr, 0, $total);
    $out[] = genchars($border, $total);
    for($a=0;$a<count($arr);$a++){
        $out[] = point2arr($arr[$a], $total);
    }
    $out[] = genchars($border, $total);
    for($a=0;$a<count($out);$a++){
        for($b=0;$b<count($out[$a]);$b++){
            echo $out[$a][$b];
        }
        echo "\n";
    }
    
}
// transform matrix 
function transformmat($arr,$total=20){
    $width = count($arr[0]);
    $c=$width-1;
    for($w=0;$w<$width;$w++){
        for($a=0;$a<count($arr);$a++){
            $ret[$c][$a] = $arr[$a][$w];
        }
        $c--;
    }
    return $ret;
}
// horizontal graphing
function print_horz_graph($arr, $total=20, $border="-", $title="", $w=""){
    if($w <> ""){
        array_reverse($arr);
        $end = count($arr)-1;
        reset($arr);
        while(list($key,$val)=each($arr)){
            $newarr[] = $arr[$key];
            $end--;
            if($end <= 0){break;}
        }
        #array_reverse($newarr);
        $arr=$newarr;
    }
    #print_R($arr);
    $bottom = "high: ".most($arr).", low: ".least($arr);
    $arr = normalize($arr, 0, $total-2);
    #$out[] = genchars($border, $total);
    for($a=0;$a<count($arr);$a++){
        $out[] = point2arr($arr[$a], $total);
    }
    $out = transformmat($out,$total);
    $ret .= genchars("-", count($out[0])+2, $title, false);
    #echo "\n";
    $ret .= "\n";
    for($a=0;$a<count($out);$a++){
        #echo termcolored("|", WHITE);
        $ret .=  " |";
        for($b=0;$b<count($out[$a]);$b++){
            $ret .= $out[$a][$b];
        }
        $ret .= "|";
        #echo termcolored("|", WHITE);
        $ret .= "\n";
    }
    $ret .= genchars("-", count($out[0])+2, $bottom, false, STR_PAD_LEFT);
    $ret .= "\n";
    return $ret;
    
}

// *EDIT* change mysql to whatever table you want to test
// this default should work for fun..
$db_list = mysql_list_tables("mysql");
while ($row = mysql_fetch_row($db_list)) {
   $tables[] = trim("{$row[0]}");
}
$testarr_z = array();
$testarr = array();
$qcount = 0;
$start = microtime(true);

while(1){
	$hw = get_term_specs();
	array_pad($testarr_z, ($hw['w'] - 6), 0);

	array_pad($testarr, ($hw['w'] - 6), 0);
	$time_start = microtime(true);
	$tab = $tables[rand(0,count($tables))];
	
	$amt = rand(500, 5000);
	$sql = "SELECT * FROM $tab LIMIT 0,$amt";
	$res = mysql_db_query("oracle", $sql);
	$qcount++;
	
	$time_end = microtime(true);
	$time = $time_end - $time_start;
	$testarr_z[] = $time;
	@mysql_free_result($res);
	
	if(count($testarr_z)> ($hw['w'] - 6)){
		array_shift($testarr_z);
	}
	$testarr = normalize($testarr_z, 1, 100);

	$testarr = remove_flatspots($testarr, false);
	$tottime = $time_end - $start;
	$qps = round($qcount/$tottime);
	echo print_horz_graph($testarr, $hw['h']-3, "-", "local mysql: $qcount queries/sec $qps", $hw['w']-1);

}

?>

Rails - ‘poor mans’ SQL cache.

Rails memcached is not very easy to introduce to a large rails installation. Memcached also chews up a lot of memory on the box and overall cached model does not work the way I needed it to. Basically, I have just a “few� queries that I needed to cache because pagination sucks just that bad in rails.
So, I built my own cache, similar to how I build them in PHP except I am not using disk cache, I am using MySQL itself to cache it’s own results

First, we need a table to hold all this info (note the ‘blob’ field)
CREATE TABLE `cacheditems` (
`id` int(11) NOT NULL auto_increment,
`cachekey` varchar(255) default NULL,
`created` datetime default NULL,
`expires` datetime default NULL,
`content` longblob,
`cachehit` int(11) NOT NULL,
PRIMARY KEY  (`id`),
KEY `cacheditems_cachekey_index` (`cachekey`),
KEY `cacheditems_created_index` (`created`),
KEY `cacheditems_expires_index` (`expires`)
)

Then we create a model called “cacheditem� which has the following functions
require 'digest/sha1'
class Cacheditem < ActiveRecord::Base

def self.checkfor(sql)
key = Digest::MD5.hexdigest(Marshal.dump(sql))
logger.info "%%% checking for key #{key}"
#logger.info "%%% checking by sql #{sql[0]}"
Cacheditem.find( :first, :conditions => [ “cachekey = ? AND expires > NOW()�, key] )
end

def self.getcached(sql)
key = Digest::MD5.hexdigest(Marshal.dump(sql))
logger.info “%%% getting by key #{key}�
#logger.info “%%% getting by sql #{sql[0]}�
getc = Cacheditem.find( :first, :conditions => [ “cachekey = ?�, key] )
hitcount = getc.cachehit + 1
Cacheditem.update(getc.id, {:cachehit => hitcount})
Cacheditem.delete_all “expires < NOW()"  # cleaner
return Marshal.load( getc.content )
end

def self.storeresult(sql, result)
key = Digest::MD5.hexdigest(Marshal.dump(sql))
logger.info "%%% storing by key #{key}"
content = Marshal.dump(result)
logger.level = (4) # this stops display in logs of the marshal data
ci = new()
ci.cachekey    = key
ci.created       = Time.now()
ci.expires        = 30.minutes.from_now() # change as needed
ci.content        = content
ci.cachehit       = 0
ci.save
return  result
end

end

Then, in application.rb I added the following function
def find_by_sql_cache(sql)
iscached = Cacheditem.checkfor(sql)
if iscached
Cacheditem.getcached(sql)
else
result = connection.select_all(sanitize_sql(sql), "#{name} Load").collect! { |record| instantiate(record) }
Cacheditem.storeresult(sql, result)
end
end

just throw “_cache� after any “find_by_sql� statement you have a need to cache and there you are.

This works very fast, very well, and doesn’t hog your memory. It cleans up after itself in the database, and perhaps it does that too much.. It would be easy to add in a standard garbage collection function which runs on a random but I felt this gave me much better stats of the actual thirty-minute cache…
If you use zabbix for monitoring your network, you can have fun graphs of cache statistics by adding the following to your zabbix_agentd.conf
UserParameter=mysql.totalcache,mysql –batch –skip-column-names -D {YOUR_DATABASE} -u{YOUR_USERNAME} -p{YOUR_PASSWORD} -e “SELECT count( * ) AS total, SUM( cachehit ) AS amount from cacheditems;� | cut -f1
UserParameter=mysql.cachehits,mysql –batch –skip-column-names -D {YOUR_DATABASE} -u{YOUR_USERNAME} -p{YOUR_PASSWORD} -e “SELECT count( * ) AS total, SUM( cachehit ) AS amount from cacheditems;� | cut -f2
« Newer Snippets
Older Snippets »
Showing 1-5 of 5 total  RSS