New functions for interacting with AWDB API: Snow Courses

[dylanbeaudette]

I've finally had some time to fully work through the functionality of the "new" (now several years old) AWDB API. The old SOAP interface still works, but it could be retired at anytime.

This PR starts with the manually visited snow courses. Two major pieces and one question:

• new functions to support AWDB REST API, in general, then specifically for SNOW network
• new cached site metadata

Question: it is fairly simple to request site metadata for a given network, do we need to keep a saved copy?

[brownag]

I've thought a little more about this.

While snow survey data / sites are available from AWDB, how different is the schema from what we currently return from fetchSCAN()?

If a new prototype function to obtain these data is going to be necessary, I think probably the snow survey sites should be in their own metadata dataset.

Currently we use the same interface for SCAN/SNOTEL/CSCAN/SNOWLITE etc. and all sites within SCAN_SNOTEL_metadata object work with fetchSCAN().

After this it will be necessary to filter sites by network before using fetchSCAN, which "breaks" the functionality that currently allows for downloading all data from a particular year or range of years. Or, it doesn't break it but it will make requests to many sites that return nothing, making it much less efficient since there are so many snow survey stations that get queried.

Alternately, perhaps fetchSCAN() could be extended such that it would work with snow survey data, perhaps with an additional network option, or ideally alongside all the other data wherever possible. We have had a long-standing todo item to generalize this functionality for AWDB, so it doesn't seem inconsistent with the plan to try and do this, but I am not personally very savvy on all the differences between snow survey and the other network types.

[dylanbeaudette]

Same here. I'll re-work this PR to focus on the snow survey data and a new wrapper for the REST API. I'm not going to attempt re-doing all of the SCAN/SNOTEL stuff here.

Main idea: the snow survey sites and data are fundamentally different than weather station data. Attempting to do everything with fetchSCAN() and associated metadata may be too complex. In fact, I don't think that the current fetchSCAN() will work with the SNOWLITE network data--maybe those can be grouped with the snow course site metadata and API wrapper.

[brownag]

Same here. I'll re-work this PR to focus on the snow survey data and a new wrapper for the REST API. I'm not going to attempt re-doing all of the SCAN/SNOTEL stuff here.

That sounds good.

In fact, I don't think that the current fetchSCAN() will work with the SNOWLITE network data--maybe those can be grouped with the snow course site metadata and API wrapper.

It may be the case that there are data elements that are a better fit for this future prototype... But, anecdotally at least most sites labeled "SNTLT" do work with fetchSCAN(). In this example 40 out of 46 sites have SCAN-like data for 2020:

library(soilDB)
sites <- SCAN_site_metadata()
x <- subset(sites, Network == "SNTLT")
f <- fetchSCAN(site.code = x$Site, year = 2020)
#> NOTE: data requested occurs in multiple timezones; returning in tz="US/Central"
#> multiple sensors per depth [site 1193] STO.I_2, STO.I_8, STO.I_20, STO.I-2_2, STO.I-2_8, STO.I-2_20
#> multiple sensors per depth [site 1235] STO.I_2, STO.I_8, STO.I_20, STO.I-2_2, STO.I-2_8, STO.I-2_20
#> * site 1333 [2020]: no lines available in input
#> multiple sensors per depth [site 1281] STO.I_2, STO.I_8, STO.I-2_2, STO.I-2_8
#> * site 1320 [2020]: no lines available in input
#> * site 1313 [2020]: no lines available in input
#> * site 1329 [2020]: no lines available in input
#> * site 1334 [2020]: no lines available in input
#> * site 1328 [2020]: no lines available in input
#> 61352 records (3.54 Mb transferred)
str(f)
#> List of 14
#>  $ SMS     :'data.frame':    2196 obs. of  9 variables:
#>   ..$ Site      : int [1:2196] 1193 1193 1193 1193 1193 1193 1193 1193 1193 1193 ...
#>   ..$ Date      : Date[1:2196], format: "2020-01-01" "2020-01-02" ...
#>   ..$ Time      : chr [1:2196] "12:00" "12:00" "12:00" "12:00" ...
#>   ..$ water_year: num [1:2196] 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
#>   ..$ water_day : int [1:2196] 93 94 95 96 97 98 99 100 101 102 ...
#>   ..$ value     : num [1:2196] 39.4 39.4 39.3 39.3 39.4 39.3 39 39.2 39.2 39.1 ...
#>   ..$ depth     : num [1:2196] 5 5 5 5 5 5 5 5 5 5 ...
#>   ..$ sensor.id : Factor w/ 3 levels "SMS.I_2","SMS.I_8",..: 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ datetime  : POSIXct[1:2196], format: "2020-01-01 14:00:00" "2020-01-02 14:00:00" ...
#>  $ STO     :'data.frame':    4836 obs. of  9 variables:
#>   ..$ Site      : int [1:4836] 1193 1193 1193 1193 1193 1193 1193 1193 1193 1193 ...
#>   ..$ Date      : Date[1:4836], format: "2020-01-01" "2020-01-02" ...
#>   ..$ Time      : chr [1:4836] "12:00" "12:00" "12:00" "12:00" ...
#>   ..$ water_year: num [1:4836] 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
#>   ..$ water_day : int [1:4836] 93 94 95 96 97 98 99 100 101 102 ...
#>   ..$ value     : num [1:4836] 0.4 0.4 0.4 0.4 0.4 0.4 0.5 0.5 0.4 0.5 ...
#>   ..$ depth     : num [1:4836] 5 5 5 5 5 5 5 5 5 5 ...
#>   ..$ sensor.id : Factor w/ 6 levels "STO.I_2","STO.I_8",..: 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ datetime  : POSIXct[1:4836], format: "2020-01-01 14:00:00" "2020-01-02 14:00:00" ...
#>  $ SAL     :'data.frame':    0 obs. of  0 variables
#>  $ TAVG    :'data.frame':    13141 obs. of  9 variables:
#>   ..$ Site      : int [1:13141] 1193 1193 1193 1193 1193 1193 1193 1193 1193 1193 ...
#>   ..$ Date      : Date[1:13141], format: "2020-01-01" "2020-01-02" ...
#>   ..$ Time      : chr [1:13141] "12:00" "12:00" "12:00" "12:00" ...
#>   ..$ water_year: num [1:13141] 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
#>   ..$ water_day : int [1:13141] 93 94 95 96 97 98 99 100 101 102 ...
#>   ..$ value     : num [1:13141] -8.6 -5.1 -9.6 -6.6 -5.4 -9.5 -7.5 -3.9 -7.2 -12.3 ...
#>   ..$ depth     : num [1:13141] NA NA NA NA NA NA NA NA NA NA ...
#>   ..$ sensor.id : Factor w/ 1 level "TAVG.D": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ datetime  : POSIXct[1:13141], format: "2020-01-01 14:00:00" "2020-01-02 14:00:00" ...
#>  $ TMIN    :'data.frame':    13141 obs. of  9 variables:
#>   ..$ Site      : int [1:13141] 1193 1193 1193 1193 1193 1193 1193 1193 1193 1193 ...
#>   ..$ Date      : Date[1:13141], format: "2020-01-01" "2020-01-02" ...
#>   ..$ Time      : chr [1:13141] "12:00" "12:00" "12:00" "12:00" ...
#>   ..$ water_year: num [1:13141] 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
#>   ..$ water_day : int [1:13141] 93 94 95 96 97 98 99 100 101 102 ...
#>   ..$ value     : num [1:13141] -21 -10.9 -15.4 -11.3 -16.8 -18.7 -11.9 -9.8 -12.2 -19.4 ...
#>   ..$ depth     : num [1:13141] NA NA NA NA NA NA NA NA NA NA ...
#>   ..$ sensor.id : Factor w/ 1 level "TMIN.D": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ datetime  : POSIXct[1:13141], format: "2020-01-01 14:00:00" "2020-01-02 14:00:00" ...
#>  $ TMAX    :'data.frame':    13141 obs. of  9 variables:
#>   ..$ Site      : int [1:13141] 1193 1193 1193 1193 1193 1193 1193 1193 1193 1193 ...
#>   ..$ Date      : Date[1:13141], format: "2020-01-01" "2020-01-02" ...
#>   ..$ Time      : chr [1:13141] "12:00" "12:00" "12:00" "12:00" ...
#>   ..$ water_year: num [1:13141] 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
#>   ..$ water_day : int [1:13141] 93 94 95 96 97 98 99 100 101 102 ...
#>   ..$ value     : num [1:13141] 0.2 -0.9 -6.3 -0.4 5.2 -1.2 -4.1 2.2 -2.3 -6.2 ...
#>   ..$ depth     : num [1:13141] NA NA NA NA NA NA NA NA NA NA ...
#>   ..$ sensor.id : Factor w/ 1 level "TMAX.D": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ datetime  : POSIXct[1:13141], format: "2020-01-01 14:00:00" "2020-01-02 14:00:00" ...
#>  $ PRCP    :'data.frame':    0 obs. of  0 variables
#>  $ PREC    :'data.frame':    1468 obs. of  9 variables:
#>   ..$ Site      : int [1:1468] 1193 1193 1193 1193 1193 1193 1193 1193 1193 1193 ...
#>   ..$ Date      : Date[1:1468], format: "2020-01-01" "2020-01-02" ...
#>   ..$ Time      : chr [1:1468] "12:00" "12:00" "12:00" "12:00" ...
#>   ..$ water_year: num [1:1468] 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
#>   ..$ water_day : int [1:1468] 93 94 95 96 97 98 99 100 101 102 ...
#>   ..$ value     : num [1:1468] 6.2 6.8 7.1 7.1 7.1 7.1 7.1 7.1 7.1 7.1 ...
#>   ..$ depth     : num [1:1468] NA NA NA NA NA NA NA NA NA NA ...
#>   ..$ sensor.id : Factor w/ 2 levels "PREC.I","PREC.I-2": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ datetime  : POSIXct[1:1468], format: "2020-01-01 14:00:00" "2020-01-02 14:00:00" ...
#>  $ SNWD    :'data.frame':    13429 obs. of  9 variables:
#>   ..$ Site      : int [1:13429] 1193 1193 1193 1193 1193 1193 1193 1193 1193 1193 ...
#>   ..$ Date      : Date[1:13429], format: "2020-01-01" "2020-01-02" ...
#>   ..$ Time      : chr [1:13429] "12:00" "12:00" "12:00" "12:00" ...
#>   ..$ water_year: num [1:13429] 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 ...
#>   ..$ water_day : int [1:13429] 93 94 95 96 97 98 99 100 101 102 ...
#>   ..$ value     : num [1:13429] 29 37 37 36 34 34 34 33 34 34 ...
#>   ..$ depth     : num [1:13429] NA NA NA NA NA NA NA NA NA NA ...
#>   ..$ sensor.id : Factor w/ 1 level "SNWD.I": 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ datetime  : POSIXct[1:13429], format: "2020-01-01 14:00:00" "2020-01-02 14:00:00" ...
#>  $ WTEQ    :'data.frame':    0 obs. of  0 variables
#>  $ WDIRV   :'data.frame':    0 obs. of  0 variables
#>  $ WSPDV   :'data.frame':    0 obs. of  0 variables
#>  $ LRADT   :'data.frame':    0 obs. of  0 variables
#>  $ metadata:'data.frame':    46 obs. of  19 variables:
#>   ..$ pedon_key     : chr [1:46] "66113" "70945" "73390" NA ...
#>   ..$ stationTriplet: chr [1:46] "1193:UT:SNTLT" "1235:NV:SNTLT" "1201:NV:SNTLT" "1239:CA:SNTLT" ...
#>   ..$ Site          : chr [1:46] "1193" "1235" "1201" "1239" ...
#>   ..$ stateCode     : chr [1:46] "UT" "NV" "NV" "CA" ...
#>   ..$ Network       : chr [1:46] "SNTLT" "SNTLT" "SNTLT" "SNTLT" ...
#>   ..$ Name          : chr [1:46] "Henrys Fork" "Smith Creek" "Tent Mtn Upper" "Althouse" ...
#>   ..$ dcoCode       : chr [1:46] "UT" "UT" "UT" "OR" ...
#>   ..$ countyName    : chr [1:46] "Summit" "Elko" "Elko" "Del Norte" ...
#>   ..$ HUC           : chr [1:46] "140401060201" "160401030403" "160401010704" "171003110302" ...
#>   ..$ Elevation_ft  : num [1:46] 10180 7590 8350 4970 7520 ...
#>   ..$ Latitude      : num [1:46] 40.9 40.4 41 42 40.5 ...
#>   ..$ Longitude     : num [1:46] -110 -116 -115 -124 -115 ...
#>   ..$ dataTimeZone  : num [1:46] -8 -8 -8 -8 -8 -8 -8 -8 -8 -8 ...
#>   ..$ shefId        : chr [1:46] "HENU1" "SMCN2" "TMUN2" "ABZO3" ...
#>   ..$ beginDate     : chr [1:46] "2012-09-11 00:00" "2013-06-27 00:00" "2011-12-23 00:00" "2013-06-25 00:00" ...
#>   ..$ endDate       : chr [1:46] "2100-01-01 00:00" "2100-01-01 00:00" "2100-01-01 00:00" "2100-01-01 00:00" ...
#>   ..$ upedonid      : chr [1:46] "S10UT043001" "S2013NV007901" "S2014NV007044" NA ...
#>   ..$ pedlabsampnum : chr [1:46] "11N0381" "14N0336" "15N0482" NA ...
#>   ..$ State         : chr [1:46] "Utah" "Nevada" "Nevada" "California" ...

[dylanbeaudette]

Yeah, I'll re-work this entire PR. It is now clear to me that the snow course data and metadata require a completely different approach. Thankfully the newer API is simpler to work with in many ways. It is more verbose, so will require some more thought before we can make the switch with SCAN, SNOTEL, SNOWLITE requests. That will be a different PR.