token handling overhaul

[JosiahParry]

I think the state of functionality of arcgislayers is stable enough that we can begin rejiggering the internals. This discussion is scoped on ensuring there is consistency in which the way that tokens are handled by the {arcgis} location services family of packages (consisting of only arcgisutils and arcgislayers, today).

Some notable issues are:

• tokens are created as httr2_token but are set as character scalars using set_auth_token()
• there is limited / no token validation on functions that require one such as arc_open() and arc_select()
• Token's are inconsistently set and should only be set using the appropriate headers via which I believe is X-Esri-Authorization: Bearer {token} (note that this has the potential to be a security issue)
• Automatic refreshing of tokens is not handled when available
  • this is tricky / problematic because we cant return the token and the result.

Improving boilerplate

I think a first step is to adapt @elipousson's arc_request() (#129 (comment)) function to be a little bit more bare bones. This would be used to create a basic request from a url that has the minimum that we need for each request: the authorization header and the the user-agent. Having a function for this would ensure that basic requirements are fulfilled.

1. From a url, create an httr2::request()
2. Set the User-Agent via arc_agent()
3. Check if a token is present and set it accordingly

arc_base_req <- function(url, token = NULL, error_call = rlang::caller_env()) {
  req <- arc_agent(httr2::request(url))
  
  # if token is not missing, check it
  if (!is.null(token)) {
    # ensure that the token is an httr2_token
    if (!rlang::inherits_only(token, "httr2_token")) {
      cli::cli_abort(
        "{.arg token} must be an {.cls httr2_token} not {.cls {class(token)}}",
        call = error_call
      )
    }
    # set the auth header
    req <- httr2::req_headers(req, "X-Esri-Authorization" = paste("Bearer", token[["access_token"]]))  
  }
  
  req
}

In addition to make sure boiler plate is handled cleanly by having a standardized base-request function, there's issue of handling tokens.

Token Generation and Storing

@mmachir first brought up the issue of managing multiple portal credentials almost a year ago #15 and this was brought up again by @elipousson in #128 (comment). Notably:

"....allow the use of multiple environmental variable names for different accounts."

Issue with environment variables

Environment variables are a good solution when we're working with API keys since the key is a scalar character. But in our case, we're working with OAuth tokens which are much more complex. OAuth tokens are represented as list objects with info on the token itself, its expiry, a refresh token and its expiry, and other metadata.

Environment variables are limiting in that they can be only scalars! So? How can we solve this? I think one way that could be use an R environment. We could create an environment in the package that is globally available:

# create a package level environment
token_env <- rlang::env()

Then we when we set a token, we would set it into the environment. The default token in the environment would be called ARCGIS_TOKEN, but we could also support any arbitrarily named tokens being set into the environment. arc_token() would then let us grab a token by name from the environment.

Help functions for the below

# helper fn to check the token is of the right class
obj_check_token <- function(token, call = rlang::caller_env()) {
  if (!rlang::inherits_only(token, "httr2_token")) {
    cli::cli_abort(
      "{.arg token} must be an {.cls httr2_token} not {.cls {class(token)}}"
    )  
  }
  invisible(token)
}

# helper function to ensure that all arguments passed to dots are named
check_dots_named <- function(dots, call = rlang::caller_env()) {
  dots_names <- names(dots)
  if (any(!nzchar(dots_names)) || is.null(dots_names)) {
    cli::cli_abort(
      "All arguments provided to {.arg ...} must be named",
      call = call
    )
  }
  invisible(dots)
}

# function to set token
set_token <- function(token, ...) {
  
  # one of the two must be provided
  if (rlang::is_missing(token) && rlang::is_missing(...)) {
    cli::cli_abort("Must provide {.arg token} or {.arg ...}")
  }
  
  # handle dots if present 
  if (rlang::dots_n(...) > 0) {
    
    # capture dots
    dots <- rlang::list2(...)
    
    # check em! 
    check_dots_named(dots)
    
    # check all tokens
    for (token in dots) {
      obj_check_token(token)  
    }
    
    # bind tokens to token environment
    rlang::env_bind(token_env, ...)
    
  }
  
  # if token arg is not missing set it
  if (!rlang::is_missing(token)) {
    # check if token is the right class
    obj_check_token(token)
    
    # set the environment ARCGIS_TOKEN
    rlang::env_bind(token_env, "ARCGIS_TOKEN" = token)
  }
}

We can then use a new version of arc_token() to fetch items out of the environment:

arc_token <- function(token = "ARCGIS_TOKEN") {
  # returns NULL if not found
  token_env[[token]]
}

Here's what this mock up looks like:

user_token <- auth_user()
client_token <- auth_client()

set_token(
  user = user_token,
  client = client_token
)

arc_token("user")
#> <httr2_token>
#> token_type: bearer
#> access_token: <REDACTED>
#> expires_at: 2024-01-25 15:59:06

arc_token("client")
#> <httr2_token>
#> token_type: bearer
#> access_token: <REDACTED>
#> expires_at: 2024-01-25 15:59:06

arc_token()
#> NULL

[JosiahParry]

@elipousson i'd love to know your thoughts on this and if it would get closer to allaying some of your qualms. This still does not get to the persisting tokens across sessions problem, but it does do a fair bit to ensuring more internal consistency and type checking.

[elipousson]

This all seems promising!

One possible addition: I noticed the other day that I could write the httr2 token object to disk as an rds file, restart the session, read the file back in, and the token still worked. You could stash the token in a user cache directory and encrypt the file using the built-in httr2 secret handling functions https://httr2.r-lib.org/reference/secrets.html

[JosiahParry]

That seems like it could definitely work! I wonder if we need helpers for this or a vignette would be sufficient?

[elipousson]

Helpers would be nice to set the directory and potentially handle cleanup if a user wanted to list saved tokens or clear out saved tokens. Something basic wrapping user_cache_dir could be sufficient https://github.com/r-lib/rappdirs

This isn’t a perfect implementation but I have some potentially similar helper functions (without secret handling) here in my sharepointr package https://github.com/elipousson/sharepointr/blob/67639b72b63903d5377e5f38fdeb0c1049104ff9/R/sp_cache.R

[JosiahParry]

Confirmed that we need to use X-Esri-Authorization: as the header for the token. This is because there are some configurations where an auth: bearer token might be used in addition to needing an esri token. If we set the esri token there, it overwrites it and doesnt work. The use cases is federated web-servers

[JosiahParry]

The major part of this has been fixed and is on CRAN. auth_user() supports non-interactive work and can a nicer alternative to auth_code() when you need to try the same script multiple times with restarting. Persisting tokens might be out of scope, but I think having a way to restore a persisted key or create on from an API Key should be possible too

[dancrocker]

I'm having difficulty generating a valid token for interactive use using the auth_client() method. I didn't want to create a new issue yet since I don't know if this is an actual bug, or user error on my part. Here are some details from my R session (of the information below has been truncated or masked intentionally):

R version 4.2.0 (2022-04-22 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

locale:
[1] LC_COLLATE=English_United States.utf8 LC_CTYPE=English_United States.utf8 LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C LC_TIME=English_United States.utf8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] arcgislayers_0.2.0 arcgisutils_0.2.0 arcgis_0.1.0 jsonlite_1.8.7 dbplyr_2.4.0 pool_1.0.1
[7] odbc_1.4.0 DBI_1.2.0 glue_1.6.2 magrittr_2.0.3 tidyselect_1.2.0 lubridate_1.9.2
[13] forcats_1.0.0 stringr_1.5.0 dplyr_1.1.4 purrr_1.0.2 readr_2.1.4 tidyr_1.3.0
[19] tibble_3.2.1 ggplot2_3.4.4 tidyverse_2.0.0

loaded via a namespace (and not attached):
[1] Rcpp_1.0.11 httr2_1.0.0 pillar_1.9.0 compiler_4.2.0 later_1.3.1 class_7.3-22
[7] tools_4.2.0 bit_4.0.5 timechange_0.2.0 lifecycle_1.0.3 gtable_0.3.4 pkgconfig_2.0.3
[13] rlang_1.1.1 cli_3.6.1 rstudioapi_0.15.0 curl_5.1.0 e1071_1.7-13 withr_2.5.0
[19] rappdirs_0.3.3 generics_0.1.3 vctrs_0.6.5 hms_1.1.3 classInt_0.4-9 bit64_4.0.5
[25] grid_4.2.0 sf_1.0-15 R6_2.5.1 jsonify_1.2.2 fansi_1.0.4 tzdb_0.4.0
[31] blob_1.2.4 units_0.8-3 scales_1.3.0 colorspace_2.1-0 KernSmooth_2.23-22 utf8_1.2.3
[37] proxy_0.4-27 stringi_1.8.2 munsell_0.5.0 RcppSimdJson_0.1.11

THIS TOKEN WORKS:

### AGOL Token (Interactive Use)
token <- auth_user(
  username = Sys.getenv("ARCGIS_USER"),
  password = Sys.getenv("ARCGIS_PASSWORD"),
  host = arc_host(),
  expiration = 60
)

paste(token)
[1] "bearer"
[2] "mzFcMRqhxzPAoRJavp2MJmTwzyt9xbZwPDaj..."
[3] "1721919274.47628"
[4] "[MY_USER_NAME]"
[5] "[PORTAL_URL]"

THIS TOKEN DOES NOT WORK:

# AGOL Token (Non-interactive Use)
token <- auth_client(
  client = Sys.getenv("ARCGIS_CLIENT_1"),
  secret = Sys.getenv("ARCGIS_SECRET_1"),
  host = arc_host(),
  expiration = 60

 paste(token)
[1] "bearer"                                                                                                                                                                                                                                                              
[2] "AAPT3NKHt6i2urmWtqOuug..."
[3] "1721922885.7097"                                                                                                                                                                                                                                                     
[4] "[PORTAL_URL]"   

API CALL AND ERROR MESSAGE:

fl_url <- "https://services1.arcgis.com/7iJyYTjCtKsZS1LR/ArcGIS/rest/services/service_#####..../FeatureServer/1"

>   agol_non_routine <- arc_read(url = fl_url)
Error in `arc_open()`:
! Status code: 498
Error : Invalid token.
Run `rlang::last_trace()` to see where the error occurred.
> rlang::last_trace(drop = FALSE)
<error/rlang_error>
Error in `arc_open()`:
! Status code: 498
Error : Invalid token.
---
Backtrace:
    ▆
 1. └─arcgislayers::arc_read(url = fl_url)
 2.   └─arcgislayers::arc_open(url = url, token = token)
 3.     └─arcgisutils::fetch_layer_metadata(url, token)
 4.       └─arcgisutils::detect_errors(meta, error_call = call)
 5.         └─rlang::abort(paste0(full_msg, collapse = ""), call = error_call)

I've tried running the non-interactive token in separate sessions within R-Studio, and also run via RScript using a batch file trigger. I get the same error either way. If you have any insight into what could be causing the token to fail, or how to troubleshoot this further, I would greatly appreciate it.

Thanks for developing this great package for R users!

[JosiahParry]

Thanks, Dan! How did you make the client ID and secret? I know I've ran into this issue in the past and @mmachir has been able to get it to work. She's out of office at the moment however

[dancrocker]

I had our GIS manager take ownership of the feature service and create them, as she has developer/admin credentials. It seemed very straightforward and she just sent me the values. I added them to my .renviron file as suggested, but called them ARCGIS_CLIENT_1 and ARCGIS_SECRET_1 in case we create addition OAuth credentials for other projects.