Background

Dr. Laura Norman (USGS) and Susan Wethington are finalizing a hummingbird study in Southeastern Arizona and Northern Sonora. They asked if I could help with the 2006 Santa Cruz County lidar to predict suitable hummingbird habitat. I had previously processed the Santa Cruz data for another project and this simply required re-running a few pre-written batch processes specific to the hummingbird survey locations.

Previously, I had difficulty with this data set on my old workstation because the format of the 2006 lidar data were in extremely small tiles, 2,000 feet on a side, with only 100,000 to 150,000 points per tile. For some perspective, the digital orthophotography derived SfM clouds I am currently generating are on the order of 1.3 to 2.6 billion points per file. Loading the sheer number of file tiles (>5,000 las files in the folder) was very cumbersome and difficult to work with with across large areas (it also caused 'seam lines' when generating bare earth models).

Before conducting the plot analyses I decided to go ahead and do a couple of additional re-processing actions with my new workstation.

First, I merged the entire Santa Cruz County dataset (all 5,000+ las files) into a single compressed LAZ file (~759 million points, 4.3 GiB). While this file is rather large (uncompressed LAS files tend to be 10-12x larger than LAZ) - the file size is typical of the point cloud data I am currently working with.

Next, I selected an 'area of interest' surrounding the hummingbird study locations in eastern Santa Cruz County centered around Patagonia lake and the Patagonia Mountains, that are covered by the 2006 lidar data. I excluded all of their other points in the Chiricahua Mountains and in Sonora, MX. After I generated the narrower study area I started to run the FUSION based models for calculating 'CoudMetrics'; this required a digital surface model to normalize the elevation heights, then clipping out all of the circular plots based on their XY GPS location (data from Susan). I used QGIS to produce the plot locations and developed their X_min Y_min X_max Y_max by subtracting or adding 30m to their position (note: the units of the lidar dataset are in FEET, so I had to use units equivalent to 30m = 98.425 ft).

All of the lidar output data are in units of FEET and in Arizona State Plane Central Projection (EPSG:102649).

Santa Cruz lidar data re-analysis

I ran LAStools 'lasinfo' to examine the file metadata and data provenance:

lasinfo I:\Santa_Cruz_County_LiDAR_2006\LAS\120_962.las
lasinfo (160429) report for I:\Santa_Cruz_County_LiDAR_2006\LAS\120_962.las
reporting all LAS header entries:
  file signature:             'LASF'
  file source ID:             0
  global_encoding:            0
  project ID GUID data 1-4:   00000008-001E-07D1-0000-000000000000
  version major.minor:        1.0
  system identifier:          ''
  generating software:        'Merrick LiDAR Processing System'
  file creation day/year:     189/2006
  header size:                227
  offset to point data:       939
  number var. length records: 3
  point data format:          1
  point data record length:   28
  number of point records:    4499
  number of points by return: 4285 212 2 0 0
  scale factor x y z:         0.001 0.001 0.001
  offset x y z:               0 0 0
  min x y z:                  962160.188 121747.572 5083.174
  max x y z:                  963999.822 121999.979 5362.024
variable length header record 1 of 3:
  reserved             43707
  user ID              'Merrick'
  record ID            101
  length after header  342
  description          'Flight line record'
variable length header record 2 of 3:
  reserved             43707
  user ID              'Merrick'
  record ID            102
  length after header  64
  description          'Index to page of point records'
variable length header record 3 of 3:
  reserved             43707
  user ID              'Merrick'
  record ID            103
  length after header  142
  description          'Thinned subset of file's points'
the header is followed by 2 user-defined bytes
reporting minimum and maximum for all LAS point record entries ...
  X           962160188  963999822
  Y           121747572  121999979
  Z             5083174    5362024
  intensity           0        255
  return_number       1          3
  number_of_returns   1          3
  edge_of_flight_line 0          0
  scan_direction_flag 0          1
  classification      5          8
  scan_angle_rank   -10         -4
  user_data          19         19
  point_source_ID   133        140
  gps_time 577513.940309 577521.492615
number of first returns:        4285
number of intermediate returns: 4
number of last returns:         4274
number of single returns:       4064
overview over number of returns of given pulse: 4064 425 10 0 0 0 0
histogram of classification of points:
             726  high vegetation (5)
            3773  keypoint (8)

These are old version LAS 1.0 file formats. The Z axis in FEET, projection is in Arizona State Plane Central (EPSG:102649). The classification scheme does not follow ASPRS Base Specification for lidar LAS file type - the bare ground points are classified as '8' while the above ground points (vegetation) are classed as '5'.

Clipping survey plots

The survey used a 30 meter radius from a GPS coordinate. In QGIS I loaded the hummingbird plot location .csv and created a shape file with a set 30m radius as buffer. This established the area from which the cloud metrics are computed.

For FUSION ClipData I subtracted the center coordinate 30 meters to set the min X and Y coordinate and added 30 meters to set the max X and Y. There are a total of 228 plots.

clipdata /dtm:F:\hummingbirds\dtm\santa_cruz_aoi.dtm /height /shape:1 F:\hummingbirds\laz\santa_cruz_aoi.las F:\hummingbirds\las\DDF.las 1066280.132 208375.3196 1066476.982 208572.17
clipdata /dtm:F:\hummingbirds\dtm\santa_cruz_aoi.dtm /height /shape:1 F:\hummingbirds\laz\santa_cruz_aoi.las F:\hummingbirds\las\CL1.las 1084645.472 182079.1614 1084842.322 182276.0118
clipdata /dtm:F:\hummingbirds\dtm\santa_cruz_aoi.dtm /height /shape:1 F:\hummingbirds\laz\santa_cruz_aoi.las F:\hummingbirds\las\CL2.las 1085134.76 182447.604 1085331.61 182644.4544
...

Full file: clip_plots.txt

LAS files zipped: hummingbird_60m_plots_las.zip

Once the *.las files were generated I ran the CloudMetrics

CloudMetrics /above:2 /new /id /minht:1 F:\hummingbirds\las\*.las F:\hummingbirds\cloud_metrics\cloud_metrics_full.csv

The executable produces a single table with all of the las files in the folder in rows.

Output: cloud_metrics_full.csv

IMPORTANT: Units are are in FEET

The statistics of the file include (see FUSION manual for full details):

Statistics	Details
Total number of returns
Count of returns by return number
Minimum
Maximum
Mean
Median	(output as 50th percentile)
Mode
Standard deviation
Variance
Coefficient of variation
Interquartile distance
Skewness
Kurtosis
AAD	(Average Absolute Deviation)
MADMedian	(Median of the absolute deviations from the overall median)
MADMode	(Median of the absolute deviations from the overall mode)
L-moments	(L1, L2, L3, L4)
L-moment skewness
L-moment kurtosis
Percentile values	(1st, 5th, 10th , 20th, 25th, 30th, 40th, 50th, 60th, 70th, 75th, 80th, 90th, 95th, 99th percentiles)
Canopy relief ratio	((mean - min) / (max – min))
Generalized means for the 2nd and 3rd power	(Elev quadratic mean and Elev cubic mean)
Percentage of first returns above a specified height	(canopy cover estimate)
Percentage of first returns above the mean height/elevation
Percentage of first returns above the mode height/elevation
Percentage of all returns above a specified height
Percentage of all returns above the mean height/elevation
Percentage of all returns above the mode height/elevation
Number of returns above a specified height / total first returns * 100
Number of returns above the mean height / total first returns * 100
Number of returns above the mode height / total first returns * 100

Bare Earth

In order to characterize the hummingbird survey locations I needed to normalize their canopy height profiles so that they were in units above ground instead of elevation (feet above mean sea level)

Note: The LAS files have the ground points classified at '8' rather than the standard class '2'.

GridSurfaceCreate /class:8 /median:3 /smooth:3 F:\hummingbirds\dtm\santa_cruz_aoi.dtm 6.56168 f f 2 0 2 2 F:\hummingbirds\laz\santa_cruz_aoi.las
GridSurfaceCreate v1.97 (FUSION v3.50) (Built on Oct  9 2015 10:10:29) DEBUG
 --Robert J. McGaughey--USDA Forest Service--Pacific Northwest Research Station
   -Using LASzip.dll (c) Martin Isenburg--Rapidlasso for LAS/LAZ file access
   LASzip.dll V2.4 r1 (build 150923)
Command line: GridSurfaceCreate  /class:8 /median:3 /smooth:3 F:\hummingbirds\dtm\santa_cruz_aoi.dtm 6.56168 f f 2 0 2 2 F:\hummingbirds\laz\santa_cruz_aoi.las
Run started: Thu Jun 23 13:12:37 2016
Using 1 data files:
   F:\hummingbirds\laz\santa_cruz_aoi.las: 66832024 pts
Surface model file produced:
   F:\hummingbirds\dtm\santa_cruz_aoi.dtm    Jun 23, 2016 @ 1:14 PM
Run finished: Thu Jun 23 13:14:21 2016 (elapsed time: 104 seconds)
Done

PDAL Ground (note: this scrip failed - need to retry)

pdal translate F:\hummingbirds\laz\santa_cruz_ground.laz F:\hummingbirds\laz\santa_cruz_ground.tif p2g --writers.p2g.grid_dist_y=3.28084 --writers.p2g.grid_dist_x=3.28084 --writers.p2g.radius=9.483 --writers.p2g.output_format=tif -writers.p2g.output_type=mean

Generating bare earth (*.laz)

I created a single LAZ file of the 2006 Santa Cruz lidar data using LAStools lasmerge. First I generated a filelist from the file directory of the las data:

i:
cd Santa_Cruz_County_LiDAR_2006\LAs
dir /b /s *.las > filelist.txt

Merge into single compressed laz

lasmerge -lof I:\Santa_Cruz_County_LiDAR_2006\LAS\filelist.txt -verbose -o I:\Santa_Cruz_County_LiDAR_2006\laz\santa_cruz.laz
merging headers took 151.658 sec. there are 759332112 points in total.
merging files took 1658.3 sec.

Next, I created a single bare earth *.laz file from the full cloud using LAStools las2las

las2las -i F:\hummingbirds\laz\santa_cruz.laz -first_only -keep_class 8 -o F:\hummingbirds\laz\santa_cruz_ground.laz

Next, I used the Docker version of PDAL to create a bare earth surface model file.

I then modified the p2g.json file which contains PDAL's 'pipeline' instructions for PDAL points2grid.

{    "pipeline": [        "/data/santa_cruz_ground.las",        {            "filename":"/data/santa_cruz_dem_xy2m_zft",            "output_format":"tif",            "output_type":"all",            "radius":"9.843",            "grid_dist_x":"3.28084",            "grid_dist_y":"3.28984",            "type": "writers.p2g"        }    ]}

In the Shell (MINGW64)

docker -run -v /c/Users/tswetnam/PDAL:/data -t pdal/pdal pdal pipeline /data/p2g.json

Generating new Canopy Height models and digital surface models using PDAL

Digital elevation model:

pdal translate F:\hummingbirds\laz\santa_cruz_aoi_ground.las F:\hummingbirds\dem_1m.tif p2g --writers.p2g.grid_dist_y=6.2808 --writers.p2g.grid_dist_x=6.2808 --writers.p2g.radius=9.54 --writers.p2g.output_format=tif -writers.p2g.output_type=idw

Potree Modeling w/ NAIP imagery

I downloaded 2007 NAIP imagery over Santa Cruz County from the USGS Data Portal.

First color the point cloud:

C:\>cd osgeo4w64\bin
C:\OSGeo4W64\bin>las2las F:\hummingbirds\laz\santa_cruz_aoi.las -o F:\hummingbirds\laz\santa_cruz_aoi_naip.las --color-source F:\hummingbirds\naip\santa_cruz_naip_stpln.tif --file-format 1.2 --color-source-scale 256 --color-source-bands 1 2 3

Then create a Potree:

C:\>cd Potree
C:\Potree>PotreeConverter.exe --overwrite -p patagonia -o F:\xampp\potree_test -d 400 --output-format LAS -a RGB -q NICE --edl-enabled --show-skybox --title patagonia --description "Patagonia lake area, Santa Cruz County, Arizona. 2006 aerial lidar colored by 2007 NAIP (~89 mil points)" --source  F:\hummingbirds\laz\santa_cruz_aoi_naip.las
== params ==
source[0]:              F:\hummingbirds\laz\santa_cruz_aoi_naip.las
outdir:                 F:\xampp\potree_test
spacing:                0
diagonal-fraction:      400
levels:                 -1
format:
scale:                  0
pageName:               patagonia
output-format:          LAS
projection:
AABB:
min: [1.00792e+06, 153,588, 3,500.64]
max: [1.09638e+06, 203,594, 6,611.07]
size: [88,457.6, 50,006, 3,110.43]
cubic AABB:
min: [1.00792e+06, 153,588, 3,500.64]
max: [1.09638e+06, 242,046, 91,958.2]
size: [88,457.6, 88,457.6, 88,457.6]
spacing calculated from diagonal: 383.032
READING:  F:\hummingbirds\laz\santa_cruz_aoi_naip.las
INDEXING: 1,000,000 points processed; 1,000,000 points written; 8.88 seconds passed
INDEXING: 2,000,000 points processed; 2,000,000 points written; 11.6 seconds passed
INDEXING: 3,000,000 points processed; 3,000,000 points written; 14.292 seconds passed
INDEXING: 4,000,000 points processed; 4,000,000 points written; 17.372 seconds passed
INDEXING: 5,000,000 points processed; 5,000,000 points written; 20.64 seconds passed
INDEXING: 6,000,000 points processed; 6,000,000 points written; 24.081 seconds passed
INDEXING: 7,000,000 points processed; 7,000,000 points written; 27.221 seconds passed
INDEXING: 8,000,000 points processed; 8,000,000 points written; 30.068 seconds passed
INDEXING: 9,000,000 points processed; 9,000,000 points written; 33.233 seconds passed
INDEXING: 10,000,000 points processed; 10,000,000 points written; 36.378 seconds passed
FLUSHING: 17.008s
INDEXING: 11,000,000 points processed; 11,000,000 points written; 56.404 seconds passed
INDEXING: 12,000,000 points processed; 12,000,000 points written; 59.51 seconds passed
INDEXING: 13,000,000 points processed; 13,000,000 points written; 62.894 seconds passed
INDEXING: 14,000,000 points processed; 14,000,000 points written; 65.902 seconds passed
INDEXING: 15,000,000 points processed; 15,000,000 points written; 68.76 seconds passed
INDEXING: 16,000,000 points processed; 16,000,000 points written; 71.629 seconds passed
INDEXING: 17,000,000 points processed; 17,000,000 points written; 74.431 seconds passed
INDEXING: 18,000,000 points processed; 18,000,000 points written; 77.755 seconds passed
INDEXING: 19,000,000 points processed; 19,000,000 points written; 80.915 seconds passed
INDEXING: 20,000,000 points processed; 20,000,000 points written; 84.206 seconds passed
FLUSHING: 20.613s
INDEXING: 21,000,000 points processed; 21,000,000 points written; 108.382 seconds passed
INDEXING: 22,000,000 points processed; 22,000,000 points written; 111.705 seconds passed
INDEXING: 23,000,000 points processed; 23,000,000 points written; 114.66 seconds passed
INDEXING: 24,000,000 points processed; 24,000,000 points written; 117.767 seconds passed
INDEXING: 25,000,000 points processed; 25,000,000 points written; 120.797 seconds passed
INDEXING: 26,000,000 points processed; 26,000,000 points written; 123.803 seconds passed
INDEXING: 27,000,000 points processed; 27,000,000 points written; 126.918 seconds passed
INDEXING: 28,000,000 points processed; 28,000,000 points written; 130.478 seconds passed
INDEXING: 29,000,000 points processed; 29,000,000 points written; 133.836 seconds passed
INDEXING: 30,000,000 points processed; 30,000,000 points written; 137.237 seconds passed
FLUSHING: 20.895s
INDEXING: 31,000,000 points processed; 31,000,000 points written; 161.425 seconds passed
INDEXING: 32,000,000 points processed; 32,000,000 points written; 164.762 seconds passed
INDEXING: 33,000,000 points processed; 33,000,000 points written; 168.014 seconds passed
INDEXING: 34,000,000 points processed; 34,000,000 points written; 171.123 seconds passed
INDEXING: 35,000,000 points processed; 35,000,000 points written; 174.081 seconds passed
INDEXING: 36,000,000 points processed; 36,000,000 points written; 177.345 seconds passed
INDEXING: 37,000,000 points processed; 37,000,000 points written; 180.349 seconds passed
INDEXING: 38,000,000 points processed; 38,000,000 points written; 183.444 seconds passed
INDEXING: 39,000,000 points processed; 39,000,000 points written; 186.777 seconds passed
INDEXING: 40,000,000 points processed; 40,000,000 points written; 190.017 seconds passed
FLUSHING: 23.256s
INDEXING: 41,000,000 points processed; 41,000,000 points written; 216.511 seconds passed
INDEXING: 42,000,000 points processed; 42,000,000 points written; 219.757 seconds passed
INDEXING: 43,000,000 points processed; 43,000,000 points written; 223.158 seconds passed
INDEXING: 44,000,000 points processed; 44,000,000 points written; 226.425 seconds passed
INDEXING: 45,000,000 points processed; 45,000,000 points written; 229.603 seconds passed
INDEXING: 46,000,000 points processed; 46,000,000 points written; 232.813 seconds passed
INDEXING: 47,000,000 points processed; 47,000,000 points written; 236.101 seconds passed
INDEXING: 48,000,000 points processed; 48,000,000 points written; 239.325 seconds passed
INDEXING: 49,000,000 points processed; 49,000,000 points written; 242.571 seconds passed
INDEXING: 50,000,000 points processed; 50,000,000 points written; 245.689 seconds passed
FLUSHING: 24.95s
INDEXING: 51,000,000 points processed; 51,000,000 points written; 273.843 seconds passed
INDEXING: 52,000,000 points processed; 52,000,000 points written; 277.072 seconds passed
INDEXING: 53,000,000 points processed; 53,000,000 points written; 280.286 seconds passed
INDEXING: 54,000,000 points processed; 54,000,000 points written; 283.479 seconds passed
INDEXING: 55,000,000 points processed; 55,000,000 points written; 286.472 seconds passed
INDEXING: 56,000,000 points processed; 56,000,000 points written; 289.577 seconds passed
INDEXING: 57,000,000 points processed; 57,000,000 points written; 292.642 seconds passed
INDEXING: 58,000,000 points processed; 58,000,000 points written; 295.688 seconds passed
INDEXING: 59,000,000 points processed; 59,000,000 points written; 298.464 seconds passed
INDEXING: 60,000,000 points processed; 60,000,000 points written; 301.412 seconds passed
FLUSHING: 19.22s
INDEXING: 61,000,000 points processed; 61,000,000 points written; 323.728 seconds passed
INDEXING: 62,000,000 points processed; 62,000,000 points written; 327.172 seconds passed
INDEXING: 63,000,000 points processed; 63,000,000 points written; 330.624 seconds passed
INDEXING: 64,000,000 points processed; 64,000,000 points written; 333.99 seconds passed
INDEXING: 65,000,000 points processed; 65,000,000 points written; 337.134 seconds passed
INDEXING: 66,000,000 points processed; 66,000,000 points written; 340.135 seconds passed
INDEXING: 67,000,000 points processed; 67,000,000 points written; 343.169 seconds passed
INDEXING: 68,000,000 points processed; 68,000,000 points written; 346.401 seconds passed
INDEXING: 69,000,000 points processed; 69,000,000 points written; 349.336 seconds passed
INDEXING: 70,000,000 points processed; 70,000,000 points written; 352.49 seconds passed
FLUSHING: 24.541s
INDEXING: 71,000,000 points processed; 71,000,000 points written; 380.341 seconds passed
INDEXING: 72,000,000 points processed; 72,000,000 points written; 383.587 seconds passed
INDEXING: 73,000,000 points processed; 73,000,000 points written; 386.924 seconds passed
INDEXING: 74,000,000 points processed; 74,000,000 points written; 390.076 seconds passed
INDEXING: 75,000,000 points processed; 75,000,000 points written; 393.131 seconds passed
INDEXING: 76,000,000 points processed; 76,000,000 points written; 396.244 seconds passed
INDEXING: 77,000,000 points processed; 77,000,000 points written; 399.122 seconds passed
INDEXING: 78,000,000 points processed; 78,000,000 points written; 402.024 seconds passed
INDEXING: 79,000,000 points processed; 79,000,000 points written; 404.982 seconds passed
INDEXING: 80,000,000 points processed; 80,000,000 points written; 407.995 seconds passed
FLUSHING: 23.381s
INDEXING: 81,000,000 points processed; 81,000,000 points written; 434.601 seconds passed
INDEXING: 82,000,000 points processed; 82,000,000 points written; 437.763 seconds passed
INDEXING: 83,000,000 points processed; 83,000,000 points written; 440.856 seconds passed
INDEXING: 84,000,000 points processed; 84,000,000 points written; 443.67 seconds passed
INDEXING: 85,000,000 points processed; 85,000,000 points written; 446.865 seconds passed
INDEXING: 86,000,000 points processed; 86,000,000 points written; 450.17 seconds passed
INDEXING: 87,000,000 points processed; 87,000,000 points written; 453.283 seconds passed
INDEXING: 88,000,000 points processed; 88,000,000 points written; 456.186 seconds passed
closing writer
conversion finished
88,879,905 points were processed and 88,879,905 points ( 100% ) were written to the output.
duration: 478.547s

Hummingbirds