Joys and tribulations of the mosaic dataset

Recently I've been really enjoying some of the benefits that the new mosaic dataset file format in ArcGIS has to offer.  For starters I usually work with large imagery collections that are too large to mosaic into a single file.  The ability to seamlessly and virtually mosaic imagery has been refreshing, and I've been satisfied with performance.  I'm also enthralled with Google Earth Engine and other cloud-based processing options, but sometimes it is nice to be able to have the data on-hand and useable within desktop GIS.

For all of the advantages of mosaic datasets there are a handful of disadvantages that are not well advertised by Esri.  We recently have encountered the message "Distributing mosaic dataset operation across four parallel instances on specified host" followed by the dreaded 99999 error.  It didn't take too long to figure out that the large raster tiles (>500 Mb) couldn't be processed on a computer with 8 Gb RAM.  When trying it on a computer with 32 Gb RAM the problem was miraculously resolved.  Lesson learned: if you are going to be building mosaic datasets do it on a computer with lots of RAM or else use many small tiles.

Our other discovery is the need for full path names.  Drive letters are almost never a problem for regular geoprocessing, but when it comes to being able to view the mosaic dataset across different computers with different mapped drive letters it becomes imperative!

New tool - Create Sampling Grid from Points for ArcGIS v. 1.0

With the advent of high-precision GPS spatially-explicit sampling designs have taken on an increasing importance in ecology and natural resource management. Spatially-explicit sampling regimes are useful for understanding processes such as attraction and repulsion that can be described using point pattern processes. This tool also opens up the possibility of random sampling within a larger grid. For example, users may want to collect field data to scale up to Landsat or MODIS pixels. It may be infeasible to collect data for an entire pixel, so some random sampling of the pixel may be necessary. Similarly there may be vegetation polygons or agricultural fields that the researcher wishes to sample in a random or a systematic manner. Finally, even if the researcher wishes to sample the entire grid having the ability to load center points or corner points onto a GPS and navigate to them may expedite field sampling. The creation of this tool was inspired by the needs of a current ongoing pygmy rabbit research project here in Nevada, Oregon, and Idaho.


This tool allows for the creation of polygons and centroids of polygons based on known points. The known points can be random locations, centroids of features of interest (e.g. polygons of agricultural fields or vegetation polygons), or regular gridded points across a landscape. This tool differs from existing tools, such as the Fishnet tools in ArcGIS, because it does not create a single grid for the entire landscape, but rather creates a local grid centered on each point in the input shapefile. It uses the following formula to achieve this:

(-1*(d/2) - 0.5) + i where d is the dimensions parameter and i is the iteration number. The two images below illustrate a grid with an even number of dimensions (6 on the left) and one with an odd number of dimensions (7 on the right). Both sampling grids are centered on points provided by the user, but the the one on the left has the original point (green) falling on a grid corner. The one on the right has the original point (provided by the user, in green) falling in the center of an individual grid tile.  The maroon points were created using the 'Feature Vertices to Points' tool, a standard ArcGIS tool with the advanced license.

Map algebra formula for calculating proportion of cover at different scales

Calculating the proportion of cover of different categorical classes (lets use a classified vegetation map as an example) isn't rocket science, however there are more steps involved that one might expect. Converting a categorical map to cover classes has a couple of advantages. One advantage is that it converts a categorical variable into a continuous variable making some models easier to develop and easier to interpret. Another advantage is that it allows the variable to be calculating at multiple spatial scales. When it comes to species distributions we usually don't expect the value of a single cell to be the primary driver of habitat selection. Rather it is the amount of different habitat types in some (usually unknown) neighborhood size that we expect to control habitat selection.

I've already reported on a tool for scaling up from categorical classes to continuous proportions of cover HERE.

In this post I provide a simple map algebra statement that can used in the raster calculator in ArcGIS to convert a single category in a categorical map to a percent cover map at some spatial scale.  This formula will do the trick:

(Float((FocalStatistics((Con("phase.tif" == 1, 1, 0)),NbrRectangle(33,33),"SUM")))) / (Float((FocalStatistics((Con("phase.tif" == 1, 1, 1)),NbrRectangle(33,33),"SUM"))))

All you need to do is swap out "phase.tif" with your raster, change the value after the == sign to represent the class that you are interested in, and change NbrRectangle(33,33) to a neighborhood size and shape of your choice.

If you have no gaps or holes in your raster dataset you are done.  However in many cases we are likely to have areas of NoData beyond which our raster will also be NoData.  To rectify this situation we can apply a second map algebra statement:

Con((IsNull("prpn.tif") == 1), 0, "prpn.tif")

Just swap out "prpn.tif" with the name of the raster resulting from the above step.  The IsNull tool ensures that NoData areas are identified and the conditional statement sets those areas to 0.

In the image below on the left the vegetation class in black is converted into a proportion cover with darker blues indicating higher values. The light blue indicates that either 1) a different vegetation class was dominant or 2) none of the vegetation classes were present in the spatial neighborhood.

Most popular downloads of 2015

Every year I tally up my papers, talks, posters, etc. for my annual evaluation. Last year I began to start tallying downloads for tools that I post online.  If my estimates are correct, I had 5,695 downloads in 2015 covering covering 26 tools/scripts. The most popular of my tools in 2015 were 1) polygon to centerline, 2) Climatic Water Deficit Toolbox, 3) Topography Toolbox (includes the Riparian Topography Toolbox and PRISM Data Helper), and 4) Landsat Toolbox with 2,034, 629, 629, and 511 downloads respectively. Hopefully, 2016 will provide opportunities for new tools and upgrades to existing ones in order to help solve research problems. All of the tools can be downloaded from HERE.

ArcGIS Idea - geoprocessing tool to create new network dataset

This one doesn't seem like rocket science to me.  It should be possible to create a new network dataset in ArcGIS in a model or with Arcpy - https://c.na9.visual.force.com/apex/ideaView?id=087E00000004nNqIAI

Geocode by Awesome

When was the last time someone sent you a long list of place names and asked you to make a map out of it?  If you are like me, the first thing that might go through your head might go like "Oh really. You couldn't have given me a lat/long??".  Geocoding in ArcGIS is a pain, because you need to have a complete geodatabase.  That might mean every river, stream, lake, street, lightpost, etc. in the USA! Type in each address into Google Maps?  No way. I'd like to get out of my office chair before the end of 2016.  BatchGeo?  Makes great maps, but you can't export the results.  Enter the Geocode by Awesome Table Add-In for Google Spreadsheets. Finally a solution that lives up to its name!

How to e-mail a shapefile

Most GIS professionals are familiar with e-mailing shapefiles.  We've been doing it for a while.  However, for folks just starting out or for those who use ArcMap the process of e-mailing a shapefile may be a little mystifying.  Nonetheless, sharing your data is one of the most valuable GIS skills.  Recently I answered this question for a colleague.

Here is what I wrote:

"It depends on the kind of information that she needs/wants.  If he/she wants both the geographic and attribute data then the best way is to take all of the files associated with the shapefile and put them in a zip file together.  At a minimum you'd need the .shp, .shx, .dbf, but often there is .proj, .shp.xml, .sbn, and even .cpg.  You'll need to use My Computer/Windows Explorer to view all of these files.  ArcGIS treats them as if they are one. The zip file keeps all files together.  It isn't a necessary step, but it is often easier than sending all of the files as individual attachments.

If she/he is just interested in the attribute data or doesn't have ArcGIS then there are two ways to go.

1.  In the attribute table select the rows by highlighting them on the left and then copy and paste into Excel.  Lately I've been finding this method to not work in 10.3, however.

2.  Open Excel.  Open the .dbf file.  Copy and paste all of the contents into a new Excel file without altering the original .dbf (or it can corrupt the shapefile).

Finally, don't forget to check the size.  Anything larger than about 27 Mb probably won't e-mail, although file size limits are specific to the e-mail service used."