Random forest and ArcScan

Recently I've been helping some colleagues in my department convert some historical maps into contour lines.  They are at the stage of using ArcScan to convert old scanned topographic maps into contour lines. Interestingly, ArcScan doesn't make use of multi-band image files, but rather single-band grayscale images. When trying to pull out contour lines as distinct from other types of lines this became problematic. Using a simple band thresholding method browns and grays tended to get confused.

The solution?  I used the Marine Geospatial Ecology Tools to perform a random forest classification to separate brown lines from all other colored lines. The results were really impressive.  Pretty much all of the gray lines were removed.  See the picture below for an illustration of the result.

Original scanned topographic map with contour lines in brown and other lines in black/gray.

The result when using a simple band thresholding approach. Black lines represent cells that will be used in the vectorization process.

The result from the random forest classification plus a small amount of speckle removal using the regiongroup tool in ArcGIS.

Historical ecology of the San Joaquin River Delta

I was having a discussion with a colleague recently and got reminded of some of the historical ecology work that has taken place in California.  Robin Grossinger's historical ecology group at the San Francisco Estuary Institute does excellent detective work.  Their work results in maps of how landscapes have changed over the past one hundred years or more and is changing how land managers actively manage lands and species.  Kudos to Robin Grossinger, Allison Whipple, Erin Beller and other historical ecologists at the San Francisco Estuary Institute for a job well done, for affecting positive change on land management practices and ecosystem restoration, and for elevating the role of historical ecology!

http://www.npr.org/2012/10/07/162393931/restore-california-delta-to-what-exactly

New versions of climatic water deficit tools and scripts

Miranda Redmond from our lab has created an R script for running PRISM climate data to generate climatic water deficit data for discrete sites. The functionality is similar to our existing spreadsheet version of the Climatic Water Deficit Toolbox for ArcGIS in that it runs on discrete sites rather than raster cells. You can view all three versions of the tools HERE as well as a whole variety of other tools. We anticipate that there will be improvements to all versions of the climatic water deficit tools in the near future so stay tuned!

An under-appreciated method for partitioning data into training and test

Perhaps one of the most under-appreciated tools in ArcGIS is subset features. While there are numerous examples of how to generate random points, which can then be sorted and used to generate fields for subsetting your data why not just do it directly? The subset features tool in the Geostatistical Analyst toolbox does this. Another alternative to check out is the Marine Geospatial Ecology Tools. MGET has numerous tools for doing modeling and is a great resource for quickly running models and bring the results back into ArcMap for display.

ArcGIS Idea - side by side catalog windows to make moving files really easy

Esri should allow side  by side catalog windows to allow for really easy file moving - http://ideas.arcgis.com/ideaView?id=087E0000000kBwrIAE

Joys and tribulations of the mosaic dataset

Recently I've been really enjoying some of the benefits that the new mosaic dataset file format in ArcGIS has to offer.  For starters I usually work with large imagery collections that are too large to mosaic into a single file.  The ability to seamlessly and virtually mosaic imagery has been refreshing, and I've been satisfied with performance.  I'm also enthralled with Google Earth Engine and other cloud-based processing options, but sometimes it is nice to be able to have the data on-hand and useable within desktop GIS.

For all of the advantages of mosaic datasets there are a handful of disadvantages that are not well advertised by Esri.  We recently have encountered the message "Distributing mosaic dataset operation across four parallel instances on specified host" followed by the dreaded 99999 error.  It didn't take too long to figure out that the large raster tiles (>500 Mb) couldn't be processed on a computer with 8 Gb RAM.  When trying it on a computer with 32 Gb RAM the problem was miraculously resolved.  Lesson learned: if you are going to be building mosaic datasets do it on a computer with lots of RAM or else use many small tiles.

Our other discovery is the need for full path names.  Drive letters are almost never a problem for regular geoprocessing, but when it comes to being able to view the mosaic dataset across different computers with different mapped drive letters it becomes imperative!

New tool - Create Sampling Grid from Points for ArcGIS v. 1.0

With the advent of high-precision GPS spatially-explicit sampling designs have taken on an increasing importance in ecology and natural resource management. Spatially-explicit sampling regimes are useful for understanding processes such as attraction and repulsion that can be described using point pattern processes. This tool also opens up the possibility of random sampling within a larger grid. For example, users may want to collect field data to scale up to Landsat or MODIS pixels. It may be infeasible to collect data for an entire pixel, so some random sampling of the pixel may be necessary. Similarly there may be vegetation polygons or agricultural fields that the researcher wishes to sample in a random or a systematic manner. Finally, even if the researcher wishes to sample the entire grid having the ability to load center points or corner points onto a GPS and navigate to them may expedite field sampling. The creation of this tool was inspired by the needs of a current ongoing pygmy rabbit research project here in Nevada, Oregon, and Idaho.


This tool allows for the creation of polygons and centroids of polygons based on known points. The known points can be random locations, centroids of features of interest (e.g. polygons of agricultural fields or vegetation polygons), or regular gridded points across a landscape. This tool differs from existing tools, such as the Fishnet tools in ArcGIS, because it does not create a single grid for the entire landscape, but rather creates a local grid centered on each point in the input shapefile. It uses the following formula to achieve this:

(-1*(d/2) - 0.5) + i where d is the dimensions parameter and i is the iteration number. The two images below illustrate a grid with an even number of dimensions (6 on the left) and one with an odd number of dimensions (7 on the right). Both sampling grids are centered on points provided by the user, but the the one on the left has the original point (green) falling on a grid corner. The one on the right has the original point (provided by the user, in green) falling in the center of an individual grid tile.  The maroon points were created using the 'Feature Vertices to Points' tool, a standard ArcGIS tool with the advanced license.