Tuesday, November 30, 2010

A Browser in Korn

Here's a quick tcpclient how-to implementing a simple web browser:

#!/usr/local/bin/ksh

printf "GET / HTTP/1.1\r\n" >&6
printf "Host: www.blogger.com\r\n" >&6
printf "User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.4) Gecko/20100630 Firefox/3.6.4\r\n" >&6
printf "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n" >&6
printf "Accept-Language: en-us,en;q=0.5\r\n" >&6
printf "Accept-Encoding: gzip,deflate\r\n" >&6
printf "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n" >&6
printf "Keep-Alive: 115\r\n" >&6
printf "Connection: close\r\n" >&6
printf "Cookie: name=value\r\n\r\n" >&6

while read -u7 input; do
        echo $input
done


Save that into a file called http.sh then issue forth the incantation:
$ /usr/local/bin/tcpclient www.blogger.com 80 ./http.sh || echo connection died

Thursday, November 11, 2010

Poor man's geocoder

Many addresses can be interpolated from the Census Bureau's TIGER/Line data.  Unfortunately, the TIGER database is woefully incomplete, right now, though the LUCA program may add a bunch of data for us.  Expect to see that release (the 2010 shapefiles) in March, 2011.  In my testing, I was able to assign tlids to anywhere from 60% to 85% of my input of DPV-confirmed addresses, so if you're looking for better coverage...  this solution is not for you.

The rough steps to geocode an address are these:

1)  Get the appropriate TIGER/Line shapefile from the Census.  You need to get the county All Lines file, as this is the one which has points with latitude and longitude.
2)  Convert the shapefile for easier access (optional); rather than deal with learning a GIS system, such as GRASS, I opted to load the shapefile data into MySQL.  I used PERL and the CPAN library called Shape to parse the shapefiles into flat files that I could load into MySQL.  I went all relational and created two tables - the first is called shape_record, and is essentially unique to TIGER/Line ID (tlid).  This table contains the ZIP Codes, street names, and address range information.  The second table is shape_detail, is related to the shape_record on tlid.  shape_detail contains each of the vertices of the TIGER/Line along with their latitude and longitude.
3)  Fairly important step, here, is to standardize the data on each side of your operation.  The field with the street name in it has some irregularities, and your input address may also need some cleaning up.
4)  Enter the shape_record table looking for your street name and ZIP code.  Narrow the result set to those tlids which have addresses which surround yours.  Note that you may have more than one tlid for a single input address.
5)  Enter the shape_detail table placing these records (generally at least two) into an array.  These are the points of your street.  Interpolate as appropriate - this part is easier expressed in code, so here's my hack-job in PHP:

        $res = mysql_query( $sql, $conn );
        if ( $res )
        {
                while ( $row = mysql_fetch_assoc( $res ) )
                {
                        $verts[$row['vertid']] = array( 'lat' => $row['lat'], 'lng' => $row['lng'] );
                }
                mysql_free_result( $res );

                // get the line segments;
                //calc total line length;
                $lines = array();
                for ( $i = 1; $i < sizeof( $verts ); $i++ )
                {
                        $lines[] = array(
                                'length' => sqrt(
                                        ( abs( $verts[$i]["lng"] - $verts[$i-1]["lng"] ) * abs( $verts[$i]["lng"] - $verts[$i-1]["lng"] ) )
                                        +
                                        ( abs( $verts[$i]["lat"] - $verts[$i-1]["lat"] ) * abs( $verts[$i]["lat"] - $verts[$i-1]["lat"] ) )
                                ),
                                'segments' => 0,
                                'verta' => $i - 1,
                                'vertb' => $i
                        );
                }
                $streetlen = 0;
                for ( $i = 0; $i < sizeof( $lines ); $i++ )
                {
                        $streetlen += $lines[$i]['length'];
                }

                // calculate the number of house-points ( tohn - fromhn )
                $segments = abs( $_REQUEST['fromhn'] - $_REQUEST['tohn'] );
                $seglen = $streetlen / $segments;
                $togo = abs( $_REQUEST['fromhn'] - $_REQUEST['number'] );

                for ( $i = 0; $i < sizeof( $lines ); $i++ )
                {
                        $lines[$i]['segments'] = $lines[$i]['length'] / $seglen;
                        $togo -= $lines[$i]['segments'];
                        if ( $togo < 0 )
                        {
                                $togo += $lines[$i]['segments'];
                                $selected_line = $i;
                                break;
                        }
                }

                //now we have a two points, establishing a line, and a third value which represents a vector along the line from the first point
                //calculate the point at the end of the vector.
                $longdiff = abs( $verts[ $lines[$i]['verta'] ]['lng'] - $verts[ $lines[$i]['vertb'] ]['lng'] );
                $latdiff = abs( $verts[ $lines[$i]['verta'] ]['lat'] - $verts[ $lines[$i]['vertb'] ]['lat'] );

                if ( $verts[ $lines[$i]['verta'] ]['lng'] > $verts[ $lines[$i]['vertb'] ]['lng'] )
                {
                        //going west
                        $longoper = -1;
                }
                else
                {
                        //going east or nowhere
                        $longoper = 1;
                }

                if( $verts[ $lines[$i]['verta'] ]['lat'] > $verts[ $lines[$i]['vertb'] ]['lat'] )
                {
                        //going south
                        $latoper = -1;
                }
                else
                {
                        //going north or nowhere
                        $latoper = 1;
                }


                if ( 0 == $longdiff * $latdiff )
                        $angle_a = 0;
                else
                        $angle_a = atan( $longdiff / $latdiff );
                $angle_b = M_PI - ( ( .5*M_PI ) + $angle_a );
                $angle_c = .5 * M_PI;
                $togo *= $seglen; //this is the hypotenuse
                $side_b = $togo * sin( $angle_b ); //this is our new latitude offset
                $side_a = $togo * sin( $angle_a ); //this is our new longitude offset

                printf( "maps.google.com/maps?z=17&q=loc:%f+%f\n",
                        $verts[ $lines[$i]['verta'] ]['lat'] + ( $side_b * $latoper ),
                        $verts[ $lines[$i]['verta'] ]['lng'] + ( $side_a * $longoper )
                );

       } 

Yes, I used straight trig, not spheroid geometry - we're talking about distances of hundreds of feet, here, so I decided that the curvature of the earth wasn't relevant.  In my test cases, I was generally within 100 feet of the actual location, and this is mostly due to the inaccuracies within the TIGER/Line data.  For example, a tlid might have a start address of 8500 and an end address of 8799, but the first house number is actually 8601.  This particular error might geocode the house a football field away from where it really sits.

Friday, November 5, 2010

C++ magic (or, How to make a bug)

I was reading through some old code of mine, something I had written 10 years ago, and came across what I thought must either be a bug I had missed or pure magic.  The essence of what I was looking at can be summarized as:

std::list< std::string > mylist;
std::string * myfunc( const char *t )
{
      mylist.push_front( t );
      std::list< std::string >::iterator i;
      i = mylist.begin();
      return( &(*i) );
}

When I read it, my gut said that push_front should barf because I'm passing a type that is not the same as the list is expecting.

This isn't actually magic, and in fact this sort of loose programming could easily lead to bugs.  The list template is calling a constructor to make it's own locally-scoped string object, and my input just happens to line up to one of the overloaded constructors for std::string class.  Observe similar behavior from the following sample program:

std::list< int > mylist
void buggy_code()
{
        mylist.push_front( 12.456F );
        std::list< int >::iterator i;
        i = mylist.begin();
        std::cout << ( *i ) << std::endl;
}

In both cases, I'm passing a quietly convertible value into the push_front() method.  While the first code would result in correct behavior, the second might easily present a bug.  The lesson to be learned, here, is that while compiler warnings are useful, they don't prevent logic bugs.

Blender procedural texturing (texture nodes) - Stained Glass

OK, I'm jumping back into Blender after an extended hiatus.  First steps, play with new stuff.

This morning, I decided to try and create a stained glass texture.

Here's the node setup:
 As you can see, the color output is mapped to color and specularity; the bump output is mapped to inverse normals.

Here's the test render:

I know it needs some work, but this could be a really nice effect.