bits and pieces

Mapping NYC GIS Data with Google Maps

April 19, 2011 In computing, Uncategorized (No comments yet)

New York City publishes lots of geographic data from a variety of city departments. A lot of it is GIS data for mapping things likes city park locations, beaches, playgrounds and bathrooms. There’s even a tree census GIS project you can download for every borough. Every street light. Zoning data. Lots of fun stuff! It’s called the NYC DataMine, the geo-data sets are here. It’s cool, but the value of the data is limited unless you’re a GIS wonk, or use GIS mapping tools, which, if you use Linux or Mac like me, you might be out of luck for the free GUI tools. Why doesn’t the city publish their data in easier to read web-formats? People could use it to throw onto Google maps, make location aware NYC applications, etc. It is possible to work with their data in this way, but it takes a little wrangling.

Let’s take a look at the city Parks GIS project.

What’s inside that zip file? It’s a set of mostly binary files that describe shapes and polygons using points and line segments that demarcate the boundaries of all the NYC parks defined in the database. The shape files are “ESRI Shapefiles“, a format created by Esri, a GIS mapping software company. According to Wikipedia, Esri has captured a lot of the GIS toolset market and apparently NYC uses their products. Along with these shape files is a DBase 3 database that contains meta-data about those shapes (like, the name of the park, what borough it’s in, it’s area, etc.). Normally, you’d open these files in a program like ArcGis, but I don’t use Windows. Besides, this is 2011. I want to look at it on the web, probably on a Google Map.

So we have a few issues. The first is that the binary ESRI Shapefile (Parks.shp) needs to be interpreted into some kind of serial format for easier handling. Libraries exist in different languages to read this file format, but I’ve found them to be a bit clunky and it’s easier just to get it into something else.

Shape files basically contain definitions of shapes identified by points in 2D space. These points are (obviously) meant to be plotted on a map. But what kind of map? How is that map projected? You remember from elementary school the basic Mercator Projection: Take a transparent globe that has a light in the middle, wrap a sheet of paper around the globe’s equator to form a cylinder, turn on the light and trace the lines being projected from the globe. (That’s why it’s called a projection, after all.) Actually, what we were all taught in elementary school is not exactly the correct physical method for creating the projection, but the point is that when you project a spherical object onto a 2D surface, it gets distorted somewhere. This is important because the points of a shapefile can be spatially referenced to any of a number of projections. Today on the web, we mostly use latitude and longitude as input to a mapping API (like Google, or Yahoo!) and let the service figure out how to flatten it out back into 2D. Points described in degrees latitude and longitude are “spherically projected” but I have found it rare indeed for GIS data to be described so simply. GIS data tends to be described in a different spatial reference, and this is where our NYC Parks data gets a little complicated.

First, we need a tool that can actually read and write shapefiles and hopefully output them into more friendly formats. This is where the Geospatial Data Abstraction Layer (GDAL) library comes in. It’s available for Debian and Ubuntu as packages, and probably most other Linux distros as well. The GDAL toolset comes with a program called ogr2ogr and that’s what we’re going to use to get the shape file into something more handy.

But, in order to effectively convert our shape file, we need to know what spatial projection the points are described in, and what we want to re-project them into. The switches for ogr2ogr we are interested in for this are -s_srs and -t_srs which identify the “source file SRS (spatial reference system)” and the SRS we want to convert/re-project into. It turns out that are a lot of ways to describe an SRS. Some are well known and organizations have labeled them in a standard way. But, often two different standards bodies or organizations use different labels for the same SRS. SRS’s are sometimes described by a formatted string of key/value pairs (sometimes called “Well Known Text” in the GIS world, or “WKT”). Some geo-spatial libraries even define their own standard for describing SRS’s (if you’ve used Proj.4 you’ll know about their way). What it comes down to is that “standards” for describing spatial references don’t really exist. Or rather, there seem to be several parallel standards. Luckily, GDAL is good at understanding them.

So what’s our input SRS? The “WKT” of that SRS is going to be found in Parks.prj (for projection?). Just cat it out:

PROJCS["NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic"],PARAMETER["False_Easting",984250.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-74.0],PARAMETER["Standard_Parallel_1",40.66666666666666],PARAMETER["Standard_Parallel_2",41.03333333333333],PARAMETER["Latitude_Of_Origin",40.16666666666666],UNIT["Foot_US",0.3048006096012192]],VERTCS["NAVD_1988",VDATUM["North_American_Vertical_Datum_1988"],PARAMETER["Vertical_Shift",0.0],PARAMETER["Direction",1.0],UNIT["Foot_US",0.3048006096012192]

Nice. So, you can see that our projection is the Lambert Conformal Conic, and we have some other parameters in here as well. As far as my research goes, things like “Datum” (“D North American 1983″) and “PROJCS” (“NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet”) indicate known US “state plane coordinate systems” that describe portions of the earth.

Ok, that’s the WKT for our input SRS (don’t worry, GDAL will just deal with that sucker). We need the output SRS. The coordinate system that GPS uses and pretty much all the mapping APIs expect as input is known as the World Geodetic System, last revised in 1984. Shorthand: WGS84. I’m not so sure ogr2ogr “knows” what WGS84 is by name. It’s man page, however, indicates that it does know about SRS’s described by a particular standards body, the Geomatics Committee. The Geomatics Committee calls WGS84 “EPSG:4326” and GDAL’s tool can handle that. (By the way, if you are using a Ruby or Python library that wraps proj.4, or you have a need to open and parse shapefile data with other tools that require quirky SRS definitions, the Geomatics Committee website has great translations of the SRS’s into WKT and proj.4 command line switches, which you will definitely need when you instantiate that RGeo object, or some such).

One more thing before actually doing this conversion/re-projection. GDAL doesn’t actually understand the Lambert Conformal Conic projection described in Parks.prj. There’s an updated (and as far as my testing goes, backwards compatible) revision of this projection which is defined as “Lambert_Conformal_Conic_2SP” and you must change your Parks.prj to read:

PROJCS["NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["False_Easting",984250.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-74.0],PARAMETER["Standard_Parallel_1",40.66666666666666],PARAMETER["Standard_Parallel_2",41.03333333333333],PARAMETER["Latitude_Of_Origin",40.16666666666666],UNIT["Foot_US",0.3048006096012192]],VERTCS["NAVD_1988",VDATUM["North_American_Vertical_Datum_1988"],PARAMETER["Vertical_Shift",0.0],PARAMETER["Direction",1.0],UNIT["Foot_US",0.3048006096012192]

OK! now… what output format do we want the shapefile in? Indeed, ogr2ogr2 can output a new ESRI shapefile, or we can do something like… output it to JSON which seems like a winning format to me (check the man page for other fun formats you can use):

ogr2ogr -f "GeoJSON" -s_srs Parks.prj -t_srs EPSG:4326 Parks.json Parks.shp

And we’re done! We have a nice JSON encoded string in Parks.json (albeit a very large one), with descriptions of all the Polygons and MultiPolygons that describe the boundaries of New York City’s parks in latitude and longitude! Easily munged to throw into a Google map or some such. Each park entry even has it’s associated meta-data.

RMagick Gem install on Debian Lenny

November 30, 2010 In computing (3 comments)

I run Debian Lenny and I want to install RMagick, a Ruby interface to the ImageMagick libraries. Let’s try it (i have the ruby1.9.1 packages installed on Lenny for using Ruby 1.9.2, so many “ruby” commands have a 1.9.1 appended to them):

sudo gem1.9.1 install rmagick

output:

...
Can't install RMagick 2.13.1. Can't find MagickWand.h.
...

Also there’s some warnings about “Found more than one ImageMagick installation.” Convential wisdom and google searching suggest that we can install that handy MagickWand.h header file dependency by installing the “libmagick9-dev” package from Lenny.  Unfortunately, if you do this and then re-install the gem, you are going to get an error that looks like:

...
checking for ImageMagick version >= 6.4.9... no
...

Ouch, so to install the MagickWand.h dependency, we had to downgrade our ImageMagick install to the point where the RMagick gem won’t even try to compile. This is the problem in how the ImageMagick Lenny packages are arranged, and I don’t quite understand the logic: “libmagic-dev” provides a more recent version of ImageMagick than “libmagic9-dev”… but only the older “libmagic9-dev” has the needed header files!

What to do? Backport a newer version of ImageMagick from Squeeze, of course! Follow these instructions for adding my Debian backports repository to your apt-sources, once you’re updated (make sure to pin anything you don’t want from my backports), do this:

sudo aptitude install libmagickwand-dev
sudo gem1.9.1 rmagick

And it compiles with the latest libmagick from Squeeze:

...
Successfully installed rmagick-2.13.1
...

New Lenny Backports

October 18, 2010 In computing (No comments yet)

After rebuilding my VirtualBox Debian Lenny development images to 64bit installations, I realized that I needed to update my Debian personal package repository to include the amd64 architecture. So! I went ahead and rebuilt the latest packages from Squeeze for amd64 and i386 for Lenny, which includes Lighttpd at 1.4.28.

The following packages from Debian Squeeze are now in the PPA:

erlang-1:14.a-dfsg-2~bpo50+1 [amd64, i386]
couchdb-0.11.0-2.1~bpo50+1 [amd64, i386]
lighttpd-1.4.28-1~bpo50+1 [amd64, i386]
dkimproxy-1.2-6~bpo50+1 [amd64, i386]

Installation instruction for aptitude

Couchdb 0.11 Backport for Lenny

May 10, 2010 In computing (One comment)

In wanting to play around with couchdb in Lenny, I found that Lenny’s official package is at version 0.8 but I wanted to test out some features from >= 0.10. Squeeze packages 0.11 so it was fairly easy to backport. I haven’t used the backport extensively, but I have noticed that Futon contains a JS error (“this.live is not a function”, futon.js?0.11.0, line 382). Perhaps this error or another issue is making the web interface essentially useless. Note that installing the backport also installs a backport of Erlang.

Installation Instructions for Aptitude:

Add this to your /etc/apt/sources.list:

deb http://www.jonmoniaci.com/debian-ppa/ lenny main contrib non-free

The install via:

aptitude update && aptitude install couchdb

UPDATE 2010-10-19:

The “this.live is not a function” problem is due to the fact that Couchdb relies on the “libjs-jquery” debian package but does not specify a version.  The jQuery “live” function was added in version 1.3.x, but Lenny ships with jQuery 1.2.x. This means you must manually install the libjs-jquery backport from either the lenny-backports repository or my repository (I have backported it in my repository — duplicating the effort of the lenny-backports folks — since I have backported a version of couchdb that relies on this package, and I think you should be able to get it all up and running with only one addition to your /etc/apt/sources.list file):

aptitude update && aptitude install libjs-jquery

Searching Common Nicknames in SOLR

May 6, 2010 In computing (No comments yet)

I’ve been using Apache SOLR 1.4 as an indexing server for search lately.  Among the fields I index are people’s names. Most of the users are English speakers, and many use their proper English name on their profiles, but their friends or colleagues only know to search for them via their common nickname. Thus, if a user stores “Kimberly” as her first name, a search for “Kim” returns no results. That’s because SOLR doesn’t know how to “stem” proper nouns. Perhaps somebody out there has written a SOLR “common English names stemmer,” but I haven’t found it.

A seemingly easy solution would be to wrap all the search queries in wildcards under the hood, so if a user enters into the search field “Kim” we silently change that to “*kim*” or some such, and only use that wildcard pattern for the “name” field in the index. For example, our query might become something like:

q=(text:kim, name:*kim*)

Where the “text” field is an aggregation of all the fields we index for each document, and the “name” field only indexes the person’s name.

Not only does this seem a little hacky, there’s a few problems. One is that I use the DisMaxRequestHandler which doesn’t allow wildcard search patterns. The other problem is that as of SOLR 1.4 leading wildcards generally don’t play nice, though I believe there are ways to handle them. Also, a search for “Richard” could never find “Dick” (or visa versa) via this method. We need something more than just simple stemming.

SOLR provides a “synonyms” token filter. This essentially allows us to create a map of words that should be considered equals. Thus, we could map:

Kim => Kimberly

SOLR would then know that “Kim” can also mean “Kimberly” and it should search for both those tokens. However, searching for “Kimberly” does not also search for “Kim”. The “=>” arrow delimiter specifies that the map is one way. By configuring the synonyms token filter to “expand” the map, designating this map:

Kim, Kimberly

(with commas) means that the synonyms work in both directions. Searching for “Kim” or “Kimberly” will search for both. And you can specify multiple synonyms in one line, like so:

Kim, Kimmy, Kimberly, Kimberlicious

Or, for one way (do you really want a search for “Kimberly” to also search for “Kimberlicious”?):

Kim, Kimmy, Kimberlicious => Kimberly

All we need now is a map file that has all the common english nicknames. UsefulEnglish.ru to the rescue! They have lists of common male and female English nicknames.

You can download my SOLR formatted synonyms file. You should note that this file doesn’t do any “one way mapping” even though it probably should in some cases. For example, based on this synonyms line:

Caroline, Carolyn, Carolina, Carlyne, Carline, Karoline, Carrie, Carry, Caddie, Caddy, Carlie, Carly, Callie, Cally, Carol, Lynn, Lynne, Lin

A search for “Carlie” also searches for “Lynn” which is probably not desirable. In reality, this line should probably be broken up into one or more maps that make a little more sense:

Carrie, Carry, Caddie, Caddy, Carlie, Carly, Callie, Cally, Carol, Caroline, Carolyn, Carlyne, Carline, Karoline

Lynn, Lynne, Lin, Caroline, Carolyn, Carlyne, Carline, Karoline

Here all the “nick names” become grouped even though the “real names” are repeated. For example, a search for “Lynn” now searches for everything on line 2… but not, say “Caddy”. And, a search for “Carolyn” will search for “Caddy” and “Lynn” since “Carolyn” appears on both lines.

I have not done any empirical benchmarking using this synonyms file, but I can say based on my observations that searches do run slower using the synonyms filter. I don’t know the exact performance implications of making the file more complicated, but I assume that length and overlapping maps/words would only serve to slow things down.

To use the synonyms filter with this file, I first created a special field type in my SOLR schema.xml file:

<fieldType name="textEnglishName" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="english_names.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
</fieldType>

Nothing fancy here! I’m using the ASCIIFoldingFilter to replace accented characters when possible with ASCII equivalents. Also, no stemming filter is present. And, the synonyms filter is only used during query, not during index.

Then, I just use the “textEnglishName” field type for any field that indexes a person’s name:

<fields>
  <field name="name" type="textEnglishName" indexed="true" stored="false"/>
</fields>

PHP Arrays, why do you hate me so?

April 18, 2010 In computing (No comments yet)

PHP array manipulation sometimes behaves oddly when you need to work with an array with “numeric” keys that are meant to behave like strings, not actual indexed positions.

In other words, you have an associative array with keys that are numeric, but you want those keys to behave like strings, not integer indexes. This often isn’t a big deal, until you have to modify that array by inserting things, which can trigger a PHP re-index, destroying your key structure.

In real life, I encountered this using Symfony 1.1 and their sfWidgetFormSelect form widget class, normally inside your form class, you add or specify this form field like so:

$this->widgetSchema['month'] = new sfWidgetFormSelect(array('choices' => self::getMonths(true)));

You can see that in this form class, I am calling a static function called “getMonths()”.  I am passing it a boolean true, here’s that static function definition:

/**
 * Returns an array for symfony select widget of months.
 * Keys are numeric, pointing to short month abbreviations
 *
 * @param bool|string $add_blank bool indicates whether to add a blank option to the array output. String indicates adding an option with a blank value, and the option text is set to the string
 * @return array
 * @access public
 */
public static function getMonths($add_blank = false)

Since this function is re-used, sometimes I want the select box to include a blank as a first option, and sometimes not depending on the form context or the form field that will include these choices. Or, if it’s not a boolean argument, i can pass in a string so the select box will have a “custom blank” field, so the value of the first choice will be a blank, but the select box option will read whatever string I pass in (like “Select Month:”, or some such).

The problem is that the months array is seen by PHP as being numerically indexed (even though I have defined the array keys with strings, not true integers). When I want to unshift, or array_merge something onto the front of the array (my “blank” option) PHP reads my array as being numerically indexed, and shifts the keys, breaking my data associations:

/**
 * Returns an array for symfony select widget of months.
 * Keys are numeric, pointing to short month abbreviations
 *
 * @param bool|string $add_blank bool indicates whether to add a blank option to the array output. String indicates adding an option with a blank value, and the option text is set to the string
 * @return array
 * @access public
 */
public static function getMonths($add_blank = false)
{
  $months = array(
    '1' => 'Jan',
    '2' => 'Feb',
    '3' => 'Mar',
    '4' => 'Apr',
    '5' => 'May',
    '6' => 'Jun',
    '7' => 'Jul',
    '8' => 'Aug',
    '9' => 'Sep',
    '10' => 'Oct',
    '11' => 'Nov',
    '12' => 'Dec'
  );

  if ($add_blank === true)
  {
    $blank = array('' => '');
  }
  elseif ($add_blank)
  {
    $blank = array('' => $add_blank);
  }
  else
  {
    $blank = false;
  }

  if (is_array($blank))
  {
    return array_merge($blank, $months);
  }
  else
  {
    return $months;
  }
}

When calling the function with a bool true parameter, this is what is returned:

array(13) {
  [""]=>
  string(0) ""
  [0]=>
  string(3) "Jan"
  [1]=>
  string(3) "Feb"
  [2]=>
  string(3) "Mar"
  [3]=>
  string(3) "Apr"
  [4]=>
  string(3) "May"
  [5]=>
  string(3) "Jun"
  [6]=>
  string(3) "Jul"
  [7]=>
  string(3) "Aug"
  [8]=>
  string(3) "Sep"
  [9]=>
  string(3) "Oct"
  [10]=>
  string(3) "Nov"
  [11]=>
  string(3) "Dec"
}

You can see that the numeric keys have been re-indexed to adjust for the added element (even though the array was defined with string representations of numeric data). An easy fix, instead of array_merge to combine the arrays, add them! (who knew?):

if (is_array($blank))
{
  return $blank + $months;
}
else
{
  return $months;
}

this returns the array as desired, without re-indexing:

array(13) {
  [""]=>
  string(0) ""
  [1]=>
  string(3) "Jan"
  [2]=>
  string(3) "Feb"
  [3]=>
  string(3) "Mar"
  [4]=>
  string(3) "Apr"
  [5]=>
  string(3) "May"
  [6]=>
  string(3) "Jun"
  [7]=>
  string(3) "Jul"
  [8]=>
  string(3) "Aug"
  [9]=>
  string(3) "Sep"
  [10]=>
  string(3) "Oct"
  [11]=>
  string(3) "Nov"
  [12]=>
  string(3) "Dec"
}

VirtualBox ACPI problems

April 18, 2010 In computing (No comments yet)

After using the freely available VMware Player for a while to launch Debian images for web development,  as well as Windows images for IE browser testing, I’ve been slowly migrating everything to VirtualBox. In my experience, VirtualBox images seem to run a bit faster and when using Windows images, the OS UI seems to be more responsive. VirtualBox doesn’t come “out-of-the-box” with virtualized network interfaces bridged to your host, so you have to do some configuration to be able to SSH or connect to the virtual images (outside of using the console provided by VirutalBox).

When I created a new VirtualBox instance (guest OS is Ubuntu Karmic) with an installation of Windows XP SP2, the image would freeze for up to 30 seconds at a time (during installation as well as normal operation after rebooting into the installed OS), and I noticed VirtualBox was logging messages like this during the freezes:

TM: Giving up catch-up attempt at a 61 452 850 245 ns lag; new total: 1 121 014 977 319 ns

After some searching, it seems like my particular Windows installation’s ACPI was not working well with VBox. This forum thread (post by user kyboren) solved the issue. It’s a long thread, so I’m re-posting below:

After booting into Windows:

• Right-click ‘My Computer’
• Go to the hardware tab, click ‘Device Manager’
• Expand the ‘Computer’ item
• Select ‘ACPI Multiprocessor PC’
• Right-click it and select ‘Update Driver’
• Choose “Install from a list or specific location (Advanced)’
• Choose ‘Don’t search. I will choose the driver to install.’
• Choose ‘Standard PC’ from the list.
• Reboot

After rebooting, Windows hardware detection will re-detect most of the virtualized hardware, simply walk through that process.

lighty 1.4.26 in debian lenny

April 1, 2010 In computing (One comment)

Recently I’ve been playing around with SWFUpload v2.2.0.  In testing, I discovered that I would always receive back from the server an HTTP 400 “Bad Request” when the uploader tried to POST.  I run Debian Lenny and Lighty. Now, Lenny ships with Lighttpd 1.4.19 and my browser has Shockwave Flash 10.0 r42 (OS 10.5, Firefox 3.6.2). Flash seems to send an erroneous “Expect: 100-Continue” HTTP header during HTTP POST operations. Apparently some servers, like Apache, will silently ignore this and allow the operation, however Lighty does not.

Before researching this more, I decided that backporting the version of Lighty from Debian Squeeze (testing) to Lenny might solve this issue. Following these instructions I built a deb package in Lenny based on source from Squeeze and installed Lighty 1.4.26.

Although the backport functions well, it seems that even though Lighty (as of version 1.4.21) allows you to set a custom configuration value to ignore the erroneous 100-Continue header, it still cannot handle the Flash mulit-part boundary bug. Alas, we have to wait until Lighty 1.5 to get this working which I’m unwilling to run in production right now.

Since others might find the backport of Lighty 1.4.26 to Lenny useful, here it is:

http://www.jonmoniaci.com/debian-ppa/pool/main/l/lighttpd/

download and install as root with:

wget http://www.jonmoniaci.com/debian-ppa/pool/main/l/lighttpd/lighttpd_1.4.26-1~bpo50+1_i386.deb
dpkg -i lighttpd_1.4.26-1~bpo50+1_i386.deb

I have not tested the other portions of the backport (mysql vhosts, etc), since I only use the main server.

UPDATE! 2010-05-07: I have recently created a Debian repository on my server so instead of just downloading and using dpkg -i to install the backport, you can add this to your /etc/apt/sources.list:

deb http://www.jonmoniaci.com/debian-ppa/ lenny main contrib non-free

Then:

aptitude update && aptitude install lighttpd

And it will install from my repository. Note that I have not signed the repository as of yet, so the packages will be considered “untrusted”.

UPDATE! 2010-10-18: The backported version of Lighttpd in the repository is now 1.4.28 (from Squeeze).

Hollow World!

April 1, 2010 In computing (No comments yet)

echo "hollow world\n";

print "Hollow World\n";

puts 'Hollow world'

#include <iostream>
 using namespace std;
 void main()
 {
   cout << "Hollow World!" << endl;
 }

document.write('<b>Hollow World</b>');