coordinates of outfalls are ignored in shp_to_inp

[ghost]

Hello all,
I tried to convert a .inp file of a sewer network (see below for the file) to a shape file but I get this error: "Error in valid_numeric_matrix(x) : !anyNA(x) is not TRUE" and the script only creates the shapefile of the polygons. I tried with all the sample files in the package and the code worked. Can you please help me? Thank you very much.
Enrico

The code that I used is the following:

if (!require(swmmr)) install.packages(swmmr)
library(tidyverse)
library(swmmr)
library(purrr)
library(dplyr)
library(sf)

inp_file <- "C:/Users/Enrico/Tesi/Paullo/R/Paullo.inp"
out_dir <- "C:/Users/Enrico/Tesi/Paullo/R"

Paullo <- read_inp(x = inp_file)
summary(Paullo)

inp_to_files(x = Paullo, name = "Paullo", path_out = out_dir)

list.files(out_dir)
c("shp", "txt", "dat") %>%
map( ~ file.path(out_dir, .)) %>%
map(list.files)

Paullo Script.txt

[dleutnant]

It seems that you have some conduits without coordinates.

library(tidyverse)
library(swmmr)

inp_file <- "/Users/dominik/Downloads/Paullo.inp"

Paullo <- read_inp(x = inp_file)
summary(Paullo)
#> 
#> ** summary of swmm model structure ** 
#> infiltration   :     horton 
#> flow_units     :        cms 
#> flow_routing   :    dynwave 
#> start_date     : 06/22/1993 
#> end_date       : 06/23/1993 
#> raingages      :          1 
#> subcatchments  :        835 
#> aquifers       :          0 
#> snowpacks      :          0 
#> junctions      :        909 
#> outfalls       :         18 
#> dividers       :          0 
#> storages       :          0 
#> conduits       :        958 
#> pumps          :          0 
#> orifices       :          0 
#> weirs          :          0 
#> outlets        :          0 
#> controls       :          0 
#> pollutants     :          0 
#> landuses       :          0 
#> lid_controls   :          0 
#> treatment      :          0 
#> *************************************

dplyr::left_join(x = Paullo$conduits, 
                 y = Paullo$coordinates, 
                 by = c(`To Node` = "Node")) %>% 
  dplyr::filter(is.na(`X-Coord`) | is.na(`Y-Coord`)) %>% 
  dplyr::select(Name, `From Node`, `To Node`, `X-Coord`, `Y-Coord`)
#> # A tibble: 18 x 5
#>    Name  `From Node` `To Node` `X-Coord` `Y-Coord`
#>    <chr> <chr>       <chr>         <dbl>     <dbl>
#>  1 C9    942         944              NA        NA
#>  2 C163  851         979              NA        NA
#>  3 C166  756         754              NA        NA
#>  4 C172  540         951              NA        NA
#>  5 C239  506         959              NA        NA
#>  6 C254  417         418              NA        NA
#>  7 C371  679         709              NA        NA
#>  8 C588  924         923              NA        NA
#>  9 C598  921         922              NA        NA
#> 10 C643  753         937              NA        NA
#> 11 C700  400         991              NA        NA
#> 12 C701  502         503              NA        NA
#> 13 C943  693         946              NA        NA
#> 14 C966  908         909              NA        NA
#> 15 C991  932         714              NA        NA
#> 16 C1001 939         943              NA        NA
#> 17 C1017 870         871              NA        NA
#> 18 C1033 771         945              NA        NA

^{Created on 2020-01-23 by the reprex package (v0.3.0)}

[ghost]

Thanks for your quick reply, the problem is related to the outfall nodes, in fact the conduits without coordinates are those that connect a node to an outfall, however from QGIS the connections are correct. I am attaching a screenshot of the shapefile relating to the outfalls in which I add the coordinate values. Thanks again
Enrico

[ghost]

I also tried to manually enter the coordinates of the outfalls in SWMM and then use the command inp_to_files and obtain the shapefiles again, but even if I use those obtained in my shp_to_inp script, the coordinates of the outfalls are not entered when I open the .inp file in SWMM. I don't know what to do, I attach the .inp file with the coordinates of the drains entered manually and the shp_to_inp script I use, thanks very much for your help,
Enrico

if (!require(swmmr)) install.packages(swmmr)
library(tidyverse)
library(swmmr)
library(purrr)
library(dplyr)
library(sf)

out_dir <- "C:/Users/Enrico/Tesi/Paullo/R"

Paullo_simulazione <- shp_to_inp(
path_options = file.path(out_dir,"txt/Paullo_options.txt"),
path_timeseries = file.path(out_dir,"dat/Paullo_timeseries_TS1.dat"),
path_polygon = file.path(out_dir,"shp/Paullo_polygon.shp"),
path_line = file.path(out_dir,"shp/Paullo_link.shp"),
path_point = file.path(out_dir,"shp/Paullo_point.shp"),
path_outfall = file.path(out_dir,"shp/Paullo_outfall.shp")
)

summary(Paullo_simulazione)

dir.create(file.path(out_dir, "Nuovo_inp"))
write_inp(Paullo_simulazione, file.path(out_dir, "Nuovo_inp", "Paullo_simulazione.inp"))

Paullo.txt

[ghost]

Sorry if I insist, probably the problem is related to the fact that I'm using version 0.9.0 of the package and not the latest version, that is 0.9.0.9000, but I can't install it since I can't find the .targz archive. I tried to edit the script of the shp_to_inp function manually with the indicated corrections relating to the junctions but it is read-only. Can someone help me? I use Rstudio and I'm not very good at programming, thank you very much.

[dleutnant]

Thanks for spotting this! Apparently, coordinates from outfalls are ignored within shp_to_inp. I need to double check this, though.

Here's a quick fix in which I i) manually extract the coordinates of the outfalls and ii) row bind them later to the other coordinates...

library(swmmr)
library(tidyverse)
library(sf)
#> Linking to GEOS 3.7.2, GDAL 2.4.2, PROJ 5.2.0

inp_file <- "/Users/dominik/Downloads/Paullo.txt"

Paullo <- read_inp(x = inp_file)

inp_to_files(x = Paullo, name = "Paullo", path_out = "/Users/dominik/Downloads/Paullo")
#> Warning: All elements of `...` must be named.
#> Did you want `geometry = c(`X-Coord`, `Y-Coord`)`?
#> Warning in abbreviate_shapefile_names(obj): Field names abbreviated for ESRI
#> Shapefile driver
#> Writing layer `Paullo_polygon' to data source `/Users/dominik/Downloads/Paullo/shp/Paullo_polygon.shp' using driver `ESRI Shapefile'
#> Writing 836 features with 22 fields and geometry type Polygon.
#> Warning: All elements of `...` must be named.
#> Did you want `geometry = c(`X-Coord`, `Y-Coord`)`?
#> Writing layer `Paullo_link' to data source `/Users/dominik/Downloads/Paullo/shp/Paullo_link.shp' using driver `ESRI Shapefile'
#> Writing 958 features with 16 fields and geometry type Line String.
#> Warning: All elements of `...` must be named.
#> Did you want `geometry = c(`X-Coord`, `Y-Coord`)`?
#> Writing layer `Paullo_point' to data source `/Users/dominik/Downloads/Paullo/shp/Paullo_point.shp' using driver `ESRI Shapefile'
#> Writing 909 features with 6 fields and geometry type Point.
#> Warning: All elements of `...` must be named.
#> Did you want `geometry = c(`X-Coord`, `Y-Coord`)`?
#> Writing layer `Paullo_outfall' to data source `/Users/dominik/Downloads/Paullo/shp/Paullo_outfall.shp' using driver `ESRI Shapefile'
#> Writing 18 features with 6 fields and geometry type Point.
#> section weirs is missing
#> section orifices is missing
#> section pumps is missing
#> section storage is missing
#> *.shp files were written to /Users/dominik/Downloads/Paullo/shp
#> *.txt file was written to /Users/dominik/Downloads/Paullo/txt
#> section curves is missing
#> timeseries.dat files were written to /Users/dominik/Downloads/Paullo/dat

out_dir <- "/Users/dominik/Downloads/Paullo"

Paullo_simulazione <- shp_to_inp(
  path_options = file.path(out_dir,"txt/Paullo_options.txt"),
  path_timeseries = file.path(out_dir,"dat/Paullo_timeseries_TS1.dat"),
  path_polygon = file.path(out_dir,"shp/Paullo_polygon.shp"),
  path_line = file.path(out_dir,"shp/Paullo_link.shp"),
  path_point = file.path(out_dir,"shp/Paullo_point.shp"),
  path_outfall = file.path(out_dir,"shp/Paullo_outfall.shp")
)

# manually get Name and coordinates from outfalls
tmp_outfalls <- file.path(out_dir,"shp/Paullo_outfall.shp") %>% 
  sf::st_read(stringsAsFactors = F) %>% 
  dplyr::bind_cols(sf::st_drop_geometry(.), 
                   tibble::as_tibble(sf::st_coordinates(.))) %>% 
  dplyr::select(Name, X,  Y) %>% 
  sf::st_drop_geometry() %>% 
  tibble::as_tibble()
#> Reading layer `Paullo_outfall' from data source `/Users/dominik/Downloads/Paullo/shp/Paullo_outfall.shp' using driver `ESRI Shapefile'
#> Simple feature collection with 18 features and 6 fields
#> geometry type:  POINT
#> dimension:      XY
#> bbox:           xmin: 530399.1 ymin: 5028844 xmax: 532649.5 ymax: 5029711
#> epsg (SRID):    NA
#> proj4string:    NA

# add outfalls to coorinates
Paullo_simulazione$coordinates  <- 
  dplyr::bind_rows(Paullo_simulazione$coordinates, tmp_outfalls)

# rename
colnames(Paullo_simulazione$coordinates) <- c("Node", "X-Coord", "Y-Coord")

# final check 
dplyr::left_join(x = Paullo_simulazione$outfalls, 
                 y = Paullo_simulazione$coordinates, 
                 by = c("Name" = "Node"))
#> # A tibble: 18 x 8
#>    Name  Elevation Type  StageData Gated RouteTo `X-Coord` `Y-Coord`
#>    <chr>     <dbl> <chr>     <int> <chr> <chr>       <dbl>     <dbl>
#>  1 S418       92.2 FREE         NA NO    " "       531990.  5029404.
#>  2 S503       92.5 FREE         NA NO    " "       531979.  5029693.
#>  3 S754       92.0 FREE         NA NO    " "       532000.  5029347.
#>  4 S951       92.1 FREE         NA NO    " "       531997.  5029372.
#>  5 S959       92.2 FREE         NA NO    " "       531989.  5029429.
#>  6 S979       94   FREE         NA NO    " "       532267.  5029371.
#>  7 S991       92.7 FREE         NA NO    " "       531975.  5029711.
#>  8 S922       93.8 FREE         NA NO    " "       530760.  5029481.
#>  9 S714       90.2 FREE         NA NO    " "       530873.  5029063.
#> 10 S943       90.7 FREE         NA NO    " "       530865.  5029088.
#> 11 S871       93   FREE         NA NO    " "       532408.  5029391.
#> 12 S709       92.1 FREE         NA NO    " "       531295.  5029077.
#> 13 S945       91.7 FREE         NA NO    " "       531948.  5028919.
#> 14 S946       91.3 FREE         NA NO    " "       531660.  5028844.
#> 15 S937       91.9 FREE         NA NO    " "       531997.  5029299.
#> 16 S944       91.8 FREE         NA NO    " "       530911.  5029182.
#> 17 S909       93   FREE         NA NO    " "       532650.  5029332.
#> 18 S923       94   FREE         NA NO    " "       530399.  5029515.

dir.create(file.path(out_dir, "Nuovo_inp"))
write_inp(Paullo_simulazione, file.path(out_dir, "Nuovo_inp", "Paullo_simulazione.inp"))

^{Created on 2020-01-27 by the reprex package (v0.3.0)}

[dleutnant]

@DoeringA Are you already "back in business"? ;)

[ghost]

Thanks a lot for the fix, I also solved the problem by inserting the outfalls into the shapefile relating to the junctions (in this way the coordinates are read) and then going to eliminate the outfalls after the function shp_to_inp in this way:

Paullo_swmm$junction <- Paullo_swmm$junction[Paullo_swmm$junction$Name != "S418",]

... although I realize that it is not very "elegant"
One last question: do you have a tar.gz archive of the latest version of the package 0.9.0.9000? because if I try to install the developer version using:

remotes :: install_github ( "dleutnant / swmmr")

and selecting the option "CRAN packages only", the installation is not completed with error:

Error: Failed to install 'swmmr' from GitHub:
   (converted from warning) cannot remove prior installation of package 'rlang'
In addition: Warning messages:
1: In untar2 (tarfile, files, list, exdir):
   skipping pax global extended headers
2: In untar2 (tarfile, files, list, exdir):
   skipping pax global extended headers

Could you kindly help me? because otherwise I can't correct the code related to the shp_to_inp function related to the number of junctions. Many thanks

Enrico

[ghost]

Hello, maybe I should open a new topic but I try to write here in case it was my mistake. I have to insert two storage elements within my network using the shp_to_inp function and to do this I started from an example .inp file contained in the package archive and using the function inp_to_files I obtained the .shp, .txt and .dat files, in which the storage elements are contained in a shapefile of point elements. For this reason I recreated the same shapefile with my data, but when using the shp_to_inp function on my network, the storage is not inserted. Reading the package manual, however, it is reported under the "path_storage" section that the storage elements must be inserted using a .txt file, for this I cannot understand how I have to do to insert them. I report my code in case I was making mistakes and the example .inp file that I used. Thanks again for the help and availability.
Enrico
Example3.txt

# aggiorno i pacchetti di R e controllo che Rtools sia attivo
library(usethis)
library(devtools)
devtools::find_rtools()
update.packages()

#installo swmmr ed attivo le librerie necessarie
if (!require(swmmr)) devtools::install_github("dleutnant/swmmr")
library(tidyverse)
library(swmmr)
library(purrr)
library(dplyr)
library(sf)

#salvo nella cartella del progetto di R
out_dir <- "C:/Users/Enrico/Tesi/Paullo"

#lancio il comando "shape to inp" del pacchetto swmmr che converte shp e file di testo in file .inp
PAULLO <- shp_to_inp(
  path_options = file.path(out_dir,"SWMM/Paullo_Opzioni.txt"),
  path_timeseries = file.path(out_dir,"PIOGGE/Paullo_TS1.dat"),
  path_polygon = file.path(out_dir,"SWMM/Paullo_polygon.shp"),
  path_line = file.path(out_dir,"SWMM/Paullo_link.shp"),
  path_point = file.path(out_dir,"SWMM/Paullo_point.shp"),
  path_outfall = file.path(out_dir,"SWMM/Paullo_outfall.shp"),
  path_pumps = file.path(out_dir,"SWMM/Paullo_pumps.shp"),
  path_pump_curve = file.path(out_dir,"SWMM/Paullo_Pcurva1.txt"),
  path_storage = file.path(out_dir,"SWMM/Paullo_storages.shp"),
  path_storage_curve = file.path(out_dir,"SWMM/Paullo_Scurva1.txt")
)

# ricavo manualmente Nomi e coordinate degli scarichi diversamente non letti dalla funzione shp_to_inp
tmp_outfalls <- file.path(out_dir,"SWMM/Paullo_outfall.shp") %>% 
  sf::st_read(stringsAsFactors = F) %>% 
  dplyr::bind_cols(sf::st_drop_geometry(.), 
                   tibble::as_tibble(sf::st_coordinates(.))) %>% 
  dplyr::select(Name, X,  Y) %>% 
  sf::st_drop_geometry() %>% 
  tibble::as_tibble()

#aggiungo le coordinate degli scarichi al file .inp creato e rinomino i campi secondo swmm
PAULLO$coordinates  <- 
  dplyr::bind_rows(PAULLO$coordinates, tmp_outfalls)
colnames(PAULLO$coordinates) <- c("Node", "X-Coord", "Y-Coord")

#controllo la struttura del file convertito
summary(PAULLO)

#salvo il file .inp in una nuova cartella nella directory di output
#dir.create(file.path(out_dir, "PAULLO_Inp"))
write_inp(PAULLO, file.path(out_dir, "PAULLO.inp"))

[ghost]

Unfortunately I have other problems to report and I apologize once again for the inconvenience. I try to explain myself by points:

1 - The package manual probably reports an error on page 12 in the "path_storage" section as a .shp file with point features is required and not a .txt file (simple writing error).

2 - Trying to insert the storage elements, these are detected but the corresponding coordinates are not loaded correctly by the shp_to_inp function, therefore I proceeded by copying the same correction code suggested by you for the outfalls.

3 - The storage elements are also not correctly identified by the read_inp function as the "summary" section reports 0 under "storages" as the following, while these are correctly inserted in the .inp file:

** summary of swmm model structure ** 
infiltration   :     horton 
flow_units     :        cms 
flow_routing   :    dynwave 
start_date     : 06/22/1993 
end_date       : 06/23/1993 
raingages      :          1 
subcatchments  :        818 
aquifers       :          0 
snowpacks      :          0 
junctions      :        908 
outfalls       :         18 
dividers       :          0 
storages       :          0 
conduits       :        955 
pumps          :          4 
orifices       :          0 
weirs          :          0 
outlets        :          0 
controls       :          0 
pollutants     :          0 
landuses       :          0 
lid_controls   :          0 
treatment      :          0 
*************************************

I report again my complete code and the .inp file complete with all the necessary data so that you can check what I have written above. Thanks again for the availability.
Enrico
PAULLO.txt

out_dir <- "C:/Users/Enrico/Tesi/Paullo"

PAULLO <- shp_to_inp(
  path_options = file.path(out_dir,"SWMM/Paullo_Opzioni.txt"),
  path_timeseries = file.path(out_dir,"PIOGGE/Paullo_TS1.dat"),
  path_polygon = file.path(out_dir,"SWMM/Paullo_polygon.shp"),
  path_line = file.path(out_dir,"SWMM/Paullo_link.shp"),
  path_point = file.path(out_dir,"SWMM/Paullo_point.shp"),
  path_outfall = file.path(out_dir,"SWMM/Paullo_outfall.shp"),
  path_pumps = file.path(out_dir,"SWMM/Paullo_pumps.shp"),
  path_pump_curve = file.path(out_dir,"SWMM/Paullo_Pcurva1.txt"),
  path_storage = file.path(out_dir,"SWMM/Paullo_storages.shp"),
)

tmp_outfalls <- file.path(out_dir,"SWMM/Paullo_outfall.shp") %>% 
  sf::st_read(stringsAsFactors = F) %>% 
  dplyr::bind_cols(sf::st_drop_geometry(.), 
                   tibble::as_tibble(sf::st_coordinates(.))) %>% 
  dplyr::select(Name, X,  Y) %>% 
  sf::st_drop_geometry() %>% 
  tibble::as_tibble()

tmp_storages <- file.path(out_dir,"SWMM/Paullo_storages.shp") %>% 
  sf::st_read(stringsAsFactors = F) %>% 
  dplyr::bind_cols(sf::st_drop_geometry(.), 
                   tibble::as_tibble(sf::st_coordinates(.))) %>% 
  dplyr::select(Name, X,  Y) %>% 
  sf::st_drop_geometry() %>% 
  tibble::as_tibble()

PAULLO$coordinates  <- 
  dplyr::bind_rows(PAULLO$coordinates, tmp_outfalls)
colnames(PAULLO$coordinates) <- c("Node", "X-Coord", "Y-Coord")

PAULLO$coordinates  <- 
  dplyr::bind_rows(PAULLO$coordinates, tmp_storages)
colnames(PAULLO$coordinates) <- c("Node", "X-Coord", "Y-Coord")

summary(PAULLO)

write_inp(PAULLO, file.path(out_dir, "PAULLO.inp"))

[ghost]

Hello again, I'm sorry to disturb but I wanted to know if by chance he had managed to find a fix to make the read_inp and the shp_to_inp functions read the number of storage in the file. Even if you try with the Example3 file contained in its package, the number of storage units is not detected. Thank you very much
Enrico
Example3.txt

[dleutnant]

The recent development version covers your reported issue. Thank you!