Too many Forth bridge commuters in baseline network

[joeytalbot]

See screenshot - this looks wrong. Do we really have 700 cyclists commuting to Edinburgh across the Forth Bridge each day?

[joeytalbot]

This is from nptscot.github.io

[Robinlovelace]

Guess: bug in overline.

Does that appear in the build you set-up Joey?

May be related: ropensci/stplanr#458

[Robinlovelace]

Suggested approach to debug this: use tar_load() to get the input data and then step through these lines line-by-line, subsetting the relevant bit and visualising the data after each stage, e.g. with mapview::mapview(rnet_subset):

npt/_targets.R

Lines 305 to 422 in 8697d9e

	rcl = rnet_commute_list
	names(rcl$fastest)[1:4] = paste0("fastest_", names(rcl$fastest)[1:4])
	names(rcl$balanced)[1:4] = paste0("balanced_", names(rcl$balanced)[1:4])
	names(rcl$quietest)[1:4] = paste0("quietest_", names(rcl$quietest)[1:4])
	names(rcl$ebike)[1:4] = paste0("ebike_", names(rcl$ebike)[1:4])
	names_combined = lapply(rcl, names) %>% unlist(use.names = FALSE)
	names_combined = names_combined[names_combined != "geometry"]
	# Saved lots of lines of code and faster:
	rnet_long = data.table::rbindlist(rcl, fill = TRUE)
	rnet_long_sfc = sf::st_sfc(rnet_long$geometry)
	rnet_long = rnet_long %>%
	select(-geometry) |>
	mutate(across(fastest_bicycle:ebike_Quietness, function(x) tidyr::replace_na(x, 0))) %>%
	as_tibble()
	rnet_long$geometry = sf::st_sfc(rnet_long_sfc, recompute_bbox = TRUE)
	rnet_long = sf::st_as_sf(rnet_long)
	sf::st_geometry(rnet_long)
	rnet_combined = overline(rnet_long, attrib = names_combined)
	rnet_combined = rnet_combined %>%
	rowwise() %>%
	mutate(Gradient = mean(fastest_Gradient, balanced_Gradient, quietest_Gradient, ebike_Gradient)) %>%
	mutate(Quietness = mean(fastest_Quietness, balanced_Quietness, quietest_Quietness, ebike_Quietness)) %>%
	select(-matches("_Q|_Gr")) %>%
	mutate(across(matches("bicycle", round_sdc))) %>%
	mutate(Gradient = round(Gradient, digits = 1))
	# see code/tests/test-quietness-network.R
	rnet_bicycle = rnet_combined %>%
	select(matches("bicycle")) %>%
	sf::st_drop_geometry()
	rnet_combined$total_cyclists_segment = rowSums(rnet_bicycle)
	names(rnet_combined)[1:8] = paste0("commute_", names(rnet_combined))[1:8]
	rnet_commute = rnet_combined %>%
	filter(total_cyclists_segment > 0) %>%
	select(-total_cyclists_segment) %>%
	as.data.frame() %>%
	sf::st_as_sf()
	# Create rnet_school
	rcl = rnet_school_list
	names(rcl$fastest)[1:4] = paste0("fastest_", names(rcl$fastest)[1:4])
	names(rcl$balanced)[1:4] = paste0("balanced_", names(rcl$balanced)[1:4])
	names(rcl$quietest)[1:4] = paste0("quietest_", names(rcl$quietest)[1:4])
	names(rcl$ebike)[1:4] = paste0("ebike_", names(rcl$ebike)[1:4])
	names_combined = lapply(rcl, names) %>% unlist(use.names = FALSE)
	names_combined = names_combined[names_combined != "geometry"]
	# Saved lots of lines of code and faster:
	rnet_long = data.table::rbindlist(rcl, fill = TRUE)
	rnet_long_sfc = sf::st_sfc(rnet_long$geometry)
	rnet_long = rnet_long %>%
	select(-geometry) |>
	mutate(across(fastest_bicycle:ebike_Quietness, function(x) tidyr::replace_na(x, 0))) %>%
	as_tibble()
	rnet_long$geometry = sf::st_sfc(rnet_long_sfc, recompute_bbox = TRUE)
	rnet_long = sf::st_as_sf(rnet_long)
	rnet_combined = overline(rnet_long, attrib = names_combined)
	rnet_combined = rnet_combined %>%
	rowwise() %>%
	mutate(Gradient = mean(fastest_Gradient, balanced_Gradient, quietest_Gradient, ebike_Gradient)) %>%
	mutate(Quietness = mean(fastest_Quietness, balanced_Quietness, quietest_Quietness, ebike_Quietness)) %>%
	select(-matches("_Q|_Gr")) %>%
	mutate(across(matches("bicycle", round_sdc))) %>%
	mutate(Gradient = round(Gradient, digits = 1))
	# see code/tests/test-quietness-network.R
	rnet_bicycle = rnet_combined %>%
	select(matches("bicycle")) %>%
	sf::st_drop_geometry()
	rnet_combined$total_cyclists_segment = rowSums(rnet_bicycle)
	names(rnet_combined)[1:8] = paste0("school_", names(rnet_combined))[1:8]
	rnet_school = rnet_combined %>%
	filter(total_cyclists_segment > 0) %>%
	select(-total_cyclists_segment) %>%
	as.data.frame() %>%
	sf::st_as_sf()
	rnet_long = list(rnet_commute, rnet_school)
	rnet_long = data.table::rbindlist(rnet_long, fill = TRUE)
	# names(rnet_long)
	# names(rnet_long)[c(1:8,10:19,9)]
	columns_to_keep = c(
	"commute_fastest_bicycle",
	"commute_fastest_bicycle_go_dutch",
	"commute_balanced_bicycle",
	"commute_balanced_bicycle_go_dutch",
	"commute_quietest_bicycle",
	"commute_quietest_bicycle_go_dutch",
	"commute_ebike_bicycle",
	"commute_ebike_bicycle_go_dutch",
	"school_fastest_bicycle",
	"school_fastest_bicycle_go_dutch",
	"school_balanced_bicycle",
	"school_balanced_bicycle_go_dutch",
	"school_quietest_bicycle",
	"school_quietest_bicycle_go_dutch",
	"school_ebike_bicycle",
	"school_ebike_bicycle_go_dutch",
	"Quietness",
	"Gradient"
	)
	rnet_long_attributes = as_tibble(rnet_long)[columns_to_keep] %>%
	mutate_all(tidyr::replace_na, 0) %>%
	as_tibble()
	rnet_long_attributes$geometry = sf::st_sfc(rnet_long$geometry, recompute_bbox = TRUE)
	rnet_long = sf::st_as_sf(rnet_long_attributes)

Sound reasonable @joeytalbot ?

[Robinlovelace]

Results for Edinburgh from latest version of the build:

[Robinlovelace]

It's there in the rnet_commute_fastest object also, before it's combined:

tar_visnetwork(T)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.2     ✔ readr     2.1.4
✔ forcats   1.0.0     ✔ stringr   1.5.0
✔ ggplot2   3.4.2     ✔ tibble    3.2.1
✔ lubridate 1.9.2     ✔ tidyr     1.3.0
✔ purrr     1.0.1     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Linking to GEOS 3.10.2, GDAL 3.4.1, PROJ 8.2.1; sf_use_s2() is TRUE
Skipping install of 'cyclestreets' from a github remote, the SHA1 (79d73f13) has not changed since last install.
  Use `force = TRUE` to force installation
 
> rnet_commute_fastest = readRDS("outputdata/v2023-06-30-12-20-48.296236_commit_25b65e550d03d725218190b22c43d0c7ed8a5217/rnet_commute_fastest.Rds")
> rnet_commute_fastest
Simple feature collection with 33636 features and 4 fields
Geometry type: LINESTRING
Dimension:     XY
Bounding box:  xmin: -3.75943 ymin: 55.85778 xmax: -3.10016 ymax: 56.16981
Geodetic CRS:  WGS 84
First 10 features:
   bicycle bicycle_go_dutch Gradient Quietness                       geometry
1        0                3        1        40 LINESTRING (-3.58654 55.877...
2        0                3        6        40 LINESTRING (-3.58714 55.877...
3        0                3        2        40 LINESTRING (-3.5924 55.8777...
4        0                3        0        50 LINESTRING (-3.58408 55.878...
5        0                3        1        50 LINESTRING (-3.58569 55.880...
6        0                3        3        50 LINESTRING (-3.58458 55.879...
7        0                3        0        40 LINESTRING (-3.58865 55.879...
8        0                3        4        40 LINESTRING (-3.58865 55.879...
11       0                3        3        40 LINESTRING (-3.62536 55.879...
12       0                3        0        60 LINESTRING (-3.61825 55.876...
> rnet_commute_fastest %>% top_n(n = 1000, weight = bicycle) %>% mapview::mapview()
Error in top_n(., n = 1000, weight = bicycle) : 
  unused argument (weight = bicycle)
> rnet_commute_fastest %>% slice_max(n = 1000, bicycle) %>% mapview::mapview()

[Robinlovelace]

From #172

[Robinlovelace]

Plan: subset all desire lines that are 20 km or less that pass through the gap and see if it matches.

[juanfonsecaLS1]

A subset of desire lines show that the cycle trips are around 75.

[Robinlovelace]

I now agree with @mem48: must be a bug in overline(), the routing code or something else downstream of the OD data.

[mem48]

I doubt it is overline, my guess the the flows are being joined to the wrong routes prior to overline. Can you recreate the map above but with routes rather than lines?

[Robinlovelace]

I doubt it is overline, my guess the the flows are being joined to the wrong routes prior to overline. Can you recreate the map above but with routes rather than lines?

Good thinking.

Next step: we need to step through the latest version of the build and see if the issue is introduced here

npt/_targets.R

Lines 134 to 137 in f1a0601

	routes_commute = get_routes(od_commute_subset,
	plans = parameters$plans, purpose = "commute",
	folder = folder_name, batch = FALSE, nrow_batch = 50000)
	routes_commute

or here

npt/_targets.R

Lines 228 to 240 in f1a0601

	rnet_commute_list = sapply(parameters$plans, function(x) NULL)
	#p = "fastest"
	for(p in parameters$plans) {
	message("Building ", p, " network")
	rnet = make_rnets(uptake_list_commute[[p]], ncores = 20)
	f = paste0("outputdata/rnet_commute_", p, ".Rds")
	saveRDS(rnet, f)
	rnet_commute_list[[p]] = rnet
	}
	# saveRDS(rnet_commute_list, "outputdata/rnet_commute_list.Rds")
	rnet_commute_list

Heads-up @joeytalbot the up-to-date inputs are ready-to-go on atumscot so please give it a go and let us know when you get a chance.

[joeytalbot]

Update on this: we can reproduce this starting with the r_commute object and running overline(r_commute$fastest, "bicycle").

Result shown below generated from commit above.

[Robinlovelace]

The OD pairs associated with the routes on that Main Street segment seem to have not relation to it:

[Robinlovelace]

Next step on this: look at reproducing the routing error starting with the csv file here https://github.com/nptscot/outputdata/releases/tag/v2023-06-30-12-20-48.296236_commit_25b65e550d03d725218190b22c43d0c7ed8a5217

[Robinlovelace]

Fixed in #175 as shown below.

2023-07-04.00-18-58.mp4

[Robinlovelace]

Next step: test it after cyclestreets/cyclestreets-r#62 was fixed.

[Robinlovelace]

Now fixed on the main branch:

[Robinlovelace]

dev branch of nptscot.io: nptscot/nptscot.github.io#66