Discrepancy between upstream comids returned by hydroloom::navigate_network_dfs() and nhdplusTools::navigate_nldi

[xingli-giam]

Hello David and hydroloom team,

I was following up on David's advice to use hydroloom::navigate_network_dfs() to obtain upstream comids for a given focal comid. The original GitHub thread is here: [https://github.com/DOI-USGS/nhdplusTools/issues/409]

I wanted to compare this approach with the comids retrieved by nldi_navigate() from the nhdplusTools package. For the first few comids I tried, hydroloom::navigate_network_dfs() returned identical comids as nhdplusTools::nldi_navigate().

However, I encountered a problem for focal comid 10010626.

library(nhdplusTools)
library(hydroloom)

# using nhdplusTools::nldi_navigate()
flowline10010626 <- navigate_nldi(list(featureSource = "comid", 
                            			featureID = 10010626), 
                           				mode = "upstreamTributaries", 
                           				distance_km = NULL)
nldi10010626 <- as.numeric(flowline10010626$UT$nhdplus_comid)
nldi10010626<-nldi10010626[order(nldi10010626)]
	length(nldi10010626)

# using David's (hydroloom::navigate_network_dfs() approach
vaa <- get_vaa() 

network <- select(vaa, comid, fromnode, tonode) |>
  add_toids(return_dendritic = FALSE)

nnd10010626 <- navigate_network_dfs(network, 10010626, direction = "up")
## see if its the same as navigate_nldi() and get_UT()
nnd10010626<-as.numeric(unlist(nnd10010626))
nnd10010626<-nnd10010626[order(nnd10010626)]
	length(nnd10010626)

# examining the discrepancy between the two approaches
comp_df<-merge(data.frame(comid=nnd10010626,nnd=1), data.frame(comid=nldi10010626,nldi=1), by='comid', all=T)
summary(comp_df)
comp_df[is.na(comp_df$nnd),]

# output
#         comid nnd nldi
# 70418 17029298  NA    1
# 70453 17029406  NA    1

# exploring why nnd is excluding comids 17029298 and 17029406 from the set of upstream watershed comids

# comid 17029298
vaa[vaa$comid %in% 17029298,] # headwater reach
# downstream hydroseq of comid 17029298 is 680029154
vaa[vaa$hydroseq%in%680029154,] # the comid of this reach is 17906169,

# the upstream comids returned by navigate_network_dfs() exclude comid 17906169
nnd10010626[nnd10010626 %in% 17906169]
# output # numeric(0)

# and unexpectedly, the upstream comids returned by nldi_navigate() also exclude comid 17906169 although it is directly downstream of comid 17029298, which is returned by nldi_navigate()
nldi10010626[nldi10010626 %in% 17906169]
# output # numeric(0)

# calling nldi_navigate() to identify upstream comids of focal comid 17906169
flowline17906169 <- navigate_nldi(list(featureSource = "comid", 
                            			featureID = 17906169), 
                           				mode = "upstreamTributaries", 
                           				distance_km = NULL)
flowline17906169 # output incorrectly excludes comid 17029298

# calling navigate_network_dfs() to identify upstream comids of focal comid 17906169
nnd17906169 <- navigate_network_dfs(network, 17906169, direction = "up")
nnd17906169 # output correctly includes comid 17029298

So the questions I have are: what does nhdplusTools::nldi_navigate() do that correctly identifies comid 17029298 as upstream of focal comid 10010626? And why is hydroloom::navigate_network_dfs() different, that it excludes 17029298 as upstream of focal comid 10010626? At the same time, the code above shows that navigate_network_dfs() is able to correctly identify 17029298 as being upstream of 10010626, but not nldi_navigate().

I'm guessing that the underlying hydrological network used by the nldi_navigate() API query is different from the VAA dataset in nhdplusTools.

I would appreciate any insights into potential reasons why the list of upstream comids might be different, and what do you all think if the more accurate set of upstream comids.

I think this is both a hydroloom and nhdplusTools issue, but I'm not sure if I can crosslist to the nhdplusTools GitHub page. Please feel free to do so if appropriate! Thanks!!

Thank you so much!

Cheers,
Xingli

[dblodgett-usgs]

This is quite a find. There appears to be a non-network diversion in the NLDI that is from the "non-network" flowline layer.

The red line is from the "non-network" table.

If I plot the flowline that comes back from the NLDI but not the local navigation I get:

Adding the rest of what comes back from the NLDI I get:

Note that little red connector is the non-network flowline. It has comid "17029406" and is in the nldi response but not in the vaa table at all!

So that's the reason. These huge navigations are uncovering a few very minor inconsistencies. I wouldn't say one is strictly right or wrong in the context of how large of a basin you are dealing with.

[xingli-giam]

Dear David, thank you for looking into this! I appreciate your work very much! I have a few follow-up questions that I would love your thoughts on: 1. What is the non-network flowline layer used in the NLDI? I couldn't find any documentation about it on the original NHDPlus v2 user guide (https://www.caee.utexas.edu/prof/maidment/giswr2015/Docs/NHDPlusV2_User_Guide.pdf). That said, I do see the non-network flowline area in my seamless flattened version of the NHDPlus v2 dataset (https://dmap-data-commons-ow.s3.amazonaws.com/NHDPlusV21/Data/NationalData/0Release_Notes_NationalData_Seamless_GeoDatabase.pdf). However, the flow direction (FLOWDIR) is listed as "Uninitialized". 1. Zooming out, I see quite a few other non-network flowlines (see attached - thick red line is the comid 17029406 that you have identified; thin red lines are other nonnetwork flowlines). There are a total of 302186 nonnetwork flowlines in the national seamless version of the NHDPlus v2, all with "Uninitialized" FLOWDIR. How does NLDI determine the direction of the diversion? I also checked the 'PLUSFLOW' table and it links COMID 17029298 to only COMID 17906169 and not the nonnetwork flowline 17029406. And in your expert opinion, do you think it is acceptable to determine upstream area (comids) just from the network flowlines, i.e., the vaa relational tables? Thank you again for your advice! I'm not sure these questions are related to either R packages, hence I am writing to you directly instead of posting them on GitHub. Let me know if you think they are relevant and I can post them to GitHub as well. Cheers, Xingli

…

--- Xingli Giam Associate Professor Department of Ecology and Evolutionary Biology University of Tennessee, Knoxville Email: ***@***.******@***.***> or ***@***.******@***.***> Web: http://www.giamlab.com Google Scholar: https://tinyurl.com/XingliGiam ResearchGate: https://www.researchgate.net/profile/Xingli_Giam Twitter: @xingli_giam<https://twitter.com/xingli_giam>

________________________________ From: David Blodgett ***@***.***> Sent: Monday, August 19, 2024 6:00 PM To: DOI-USGS/hydroloom ***@***.***> Cc: Giam, Xingli ***@***.***>; Author ***@***.***> Subject: Re: [DOI-USGS/hydroloom] Discrepancy between upstream comids returned by hydroloom::navigate_network_dfs() and nhdplusTools::navigate_nldi (Issue #42) Closed #42<#42> as completed. — Reply to this email directly, view it on GitHub<#42 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKZ4GTGQYWDK2XRMK3FKXXDZSJTHBAVCNFSM6AAAAABMYNNCXWVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTHEZTMMZSHE3DMNQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>