Missing transfer for "unused" stops

[t2gran]

Description

A normal use-case is that a train changes platform(stop) using a real-time update. This works fine, normally. But, if the new stop does not have any trips using it, a problem occurs. OTP pre-compute transfers before starting the router. But, only transfers witch is "optimal" is added to the list of regular transfers. So, when the update happens the new stop is an island - with no transfers in/out of it. In the test data below this happens at "Arna" railway station. The new stop("NSR:Quay:107764") is not used by any other trip in the system. The trip search from Bergen to Voss works fine until the update and platform change is applied. Then, no results are found.

Version of OTP used (exact commit hash or JAR name)

dev-2.x with #5460 - The file loader is needed to load the Siri ET update.

Data sets in use (links to GTFS and OSM PBF files)

The dataset is edited to a minimum, this make it easier to debug what is happening.

Transit data:
netex.zip

Update:
et-update-arna-platform1-to-3.xml.zip

No OSM data required to test this.

Router config and graph build config JSON

OTP Config

{
  "otpFeatures": {
    "ActuatorAPI": true,
    "FlexRouting": true,
    "FloatingBike": true,
    "HackSorlandsbanen": true,
    "OptimizeTransfers": true,
    "ParallelRouting": false,
    "ReportApi": true,
    "RestAPIPassInDefaultConfigAsJson": true,
    "SandboxAPITransmodelApi": true,
    "SandboxAPITravelTime": true,
    "TransferConstraints": true,
    "VehicleToStopHeuristics": true
  }
}

OTP Build Config

{
  "transitServiceStart" : "2023-10-26",
  "transitServiceEnd" : "2023-12-30",
  "dataImportReport": true,
  "areaVisibility": true,
  "platformEntriesLinking": true,
  "matchBusRoutesToStreets": false,
  "osmDefaults": {
    "osmTagMapping": "norway",
    "timeZone": "Europe/Oslo"
  },
  "maxTransferDuration": "30m",
  "netexDefaults": {
    "feedId": "EN",
    "sharedFilePattern": "tiamat-export.*\\.xml",
    "sharedGroupFilePattern": "_(\\w{3})(_flexible)?_shared_data.xml",
    "groupFilePattern": "(\\w{3})_.*\\.xml"
  },
  "streetGraph": "streetGraph-${otp.serialization.version.id}.obj",
  "graph": "graph-${otp.serialization.version.id}.obj",
  "buildReportDir": "report"
}

OTP Router Config

{
  "server": {
    "apiProcessingTimeout": "20m",
  },
  "transmodelApi": {
    "hideFeedId": true
  },
  "timetableUpdates": {
    "purgeExpiredData": false
  },
  "updaters": [
    // SIRI ET updater
    {
      "type": "siri-et-updater",
      "frequency": "5s",
      "previewInterval": "3h",
      "url": "file:///",
      "feedId": "EN",
      "blockReadinessUntilInitialized": true,
      "fuzzyTripMatching": true
    }
  ]
}

Steps to reproduce the problem

1. Start OTP
2. Run query Transmodel GraphQL query. The expected result is one itinerary with realtime=false.

Query

{
  trip(
    from: {
      place:"NSR:StopPlace:59983"
      name:"Bergen"
    },
    to: {
      place: "NSR:StopPlace:59958"
      name:"Voss"
    },
    dateTime: "2023-10-28T20:00:00+02:00",
    searchWindow:600
    itineraryFilters : {
      debug :listAll
    }
  ) {
    tripPatterns {
      legs {
        expectedStartTime
        expectedEndTime
        mode
        line {
          id
          publicCode
        }
        serviceJourneyEstimatedCalls {
          realtime
          quay {id name publicCode}
          serviceJourney { id }
          aimedDepartureTime
          expectedDepartureTime
          predictionInaccurate
        }
      }
      systemNotices {
        tag
        text
      }
    }
}

}

3. Now apply the update .
4. Run query again. The result is now empty.
5. Edit the VYG_VYG-Line-45_R40_Bergen-Voss-Myrdal-Al.xml file, by including the last commented out ServiceJourney id="SJ-1".
6. Repeat step 1. to 4. and the expected result is that it now finds the transfer and the itinerary with the with realtime=true.

[leonardehrenfried]

Are you aware of the feature flag ConsiderPatternsForDirectTransfers. It does something similar to what you need.

[t2gran]

The ConsiderPatternsForDirectTransfers : false will make OTP work (I have tested it). But, this is not a good solution. Having features witch does not support RT updates are probably not a good idea.

We are going to do a performance test to see the effect of turning ConsiderPatternsForDirectTransfers off. The real fix to this problem is to generate transfers if change in a pattern visit a new stop and that stop is not serving any other patterns/does not have any transers.

[t2gran]

This is related to issue #4040 and PR #4421. The ConsiderPatternsForDirectTransfers was added in this PR and the logic was improved. But, if this is not improving the performance, then it is probably not worth it.

[t2gran]

The number of stops/transfers after building with ConsiderPatternsForDirectTransfers turned off/on:

"ConsiderPatternsForDirectTransfers" : true
DirectTransferGenerator: 1m16s
Transfers:   598 436
Stops:        95 268

"ConsiderPatternsForDirectTransfers" : false
DirectTransferGenerator: 24s
Transfers:  2 642 715
Stops:         99 198

I also run 5000 requests for each of the configurations, but the results wary too much to be useful. When turned on, it is faster but not by much.

[slvlirnoff]

@t2gran it's interesting how the feature seems to affect the number of stops, but I guess it's the number of stops related to a transfer that is displayed?

In general the transfer "pruning" has some edge cases that aren't properly working (potentially excluding optimal solution) even without realtime, if the patterns have some kind of geographical loop.

I recall from my experimentation that it impacts performance, with the default transfer maximum duration of roughly 45mn, all stops likely connect to all stops in a city. Keeping the "best" transfers between patterns can really reduce transfer expansion and the transfer optimisation step eventually helps picking the best transfer point.

I feel that adapting relevant transfer when processing realtime shouldn't be too hard also, aren't we already doing it for realtime added pattern (gtfs timetable snapshot source)? The list of patterns added by realtime is known.

Maybe instead of pruning transfer at build time, a subset of all transfers could be selected for a given day of operation in the transit layer and updated depending on realtime data.

[t2gran]

Maybe instead of pruning transfer at build time, a subset of all transfers could be selected for a given day of operation in the transit layer and updated depending on realtime data.

Yes, this one possibility, not sure if keeping this for each day will be necessary. But, we can do this is steps. The first step will be to add transfers for realtime changes. If this is inefficient we can cashe the transfers from build-time.

[slvlirnoff]

The last publication here is also relevant to this question (Delay-Robustness shortcuts (e.g. transfers pruning/constraint)) https://github.com/kit-algo/ULTRA

[BredeD]

This use case will likely happen in ferry traffic due to weather conditions or closed roads. The ferry operator will cancel the planned services, and new extraCalls will be sent in SIRI ET. The new ones can include new stopPlaces/quays that are not in any planned data.

Norled is working to send this correctly in SIRI.

[t2gran]

There are several errors here:

1. The travel-time is not included when selecting the optimal transfer from a stop to a given pattern. For example with stop A, B and C, Route B-C, and transfers A-B 30s, and A-C 40s: The best transfer, is not necessary A-B. If the Route A-B uses 1 minute from B to C. We can simply miss the connection if we walk to B.
2. When a pattern changes RT all stops (new/dropped) needs to be checked for transfers to other patterns in both directions.

[t2gran]

Performance at Entur (STATING w/request replayer) for ConsiderPatternsForDirectTransfers OFF vs ON.

• The slow transition is caused by the deployment roll-out (47 nodes)
• The lines are from top to bothom (only the 3 relevant lines) :
  • mc raptor (yellow)
  • transit data creation (red)
  • reverse heuristic (light blue)

We clearly see that the transit data creation and heuristic suffer from the increased number of transfers.

[miklcct]

I think I have hit this bug in my National Rail deployment. I have created a lot of "fake" platforms, not used in the timetabled data but only for real-time information, to support a particular use-case of hiding "provisional" platform numbers.

The journey planner no longer returns the normal route that most people will use, because the train service is "modified" to use a platform not used in the timetable.