Memory efficient replay route #6636

kmadsen · 2022-11-19T01:45:00Z

Description

This adds a ReplayRouteSession that is able to replay super large routes at high frequencies, without running out of memory. It also integrates it into drop-in-ui, so that we can all use it!

How it works. The developer pushes events into a replayer, and the replayer simulates the events in real time. The current replayer will take all events from a route and hold them in memory. The new ReplayRouteSession will only replay decodeMinDistance events at a time. For example, if you create a 1000 kilometers route and set the decodeMinDistance to 10 kilometers. You can expect ReplayRouteSession to only hold ~10km of events in memory at a time.

Api explanation

This object is experimental but for the intents and purposes it will become permanent. The details under the cover are subject to change, but the top interface is going to be set in stone.

// When enabled, mapbox navigation will automatically drive the active route
class ReplayRouteSession : MapboxNavigationObserver {
  // Enable route replay
  fun onAttached(mapboxNavigation)

  // Disable route replay
  fun onDetached(mapboxNavigation)

  // Get the current options
  fun getOptions(): ReplayRouteSessionOptions)

  // Change the options
  fun setOptions(options: ReplayRouteSessionOptions)
}

Known issue

The ReplayRouteMapper was designed to simulate events from A to B, where the starting speed and the ending speed is 0.0 meters per second. ReplayRouteSession is using the ReplayRouteMapper to simulate each section of the route, so the driver will appear to stop and speed up again. The larger the decodeMinDistance, the less often this will happen.

Also note that this issue is fixable but will require more time and effort.

Here are memory approximations to help you pick frequency and decodeMinDistance values. Duration is more accurate for determining memory sizes, but duration requires more resources to calculate than distance. The parameters may be removed after the internal improvements are completed.

What kind of memory improvement to expect

Old formulas to determine memory requirements

event_size ~500-300 bytes
memory_required = (duration * frequency) * event_size

New formulas

event_size ~500-300 bytes
memory_required = (decodeMinDistance * frequency) * event_size

In other words: When duration becomes 40 hours. The memory requirement can be clamped to 30 minutes.

codecov · 2022-11-23T02:24:17Z

Codecov Report

Merging #6636 (263c216) into main (1a7522f) will increase coverage by 0.03%.
The diff coverage is 81.81%.

@@             Coverage Diff              @@
##               main    #6636      +/-   ##
============================================
+ Coverage     72.31%   72.35%   +0.03%     
- Complexity     5323     5369      +46     
============================================
  Files           753      755       +2     
  Lines         29089    29250     +161     
  Branches       3448     3476      +28     
============================================
+ Hits          21037    21164     +127     
- Misses         6667     6681      +14     
- Partials       1385     1405      +20

Impacted Files	Coverage Δ
.../navigation/core/replay/route/ReplayRouteDriver.kt	`85.36% <16.66%> (-5.55%)`	⬇️
...igation/dropin/tripsession/TripSessionComponent.kt	`84.44% <40.00%> (ø)`
...navigation/core/replay/route/ReplayRouteSession.kt	`71.60% <71.60%> (ø)`
...gation/core/replay/history/ReplayEventSimulator.kt	`85.91% <90.00%> (+1.60%)`	⬆️
...ion/core/replay/route/ReplayRouteSessionOptions.kt	`95.45% <95.45%> (ø)`
...on/core/replay/route/ReplayPolylineDecodeStream.kt	`97.14% <97.14%> (ø)`
...om/mapbox/navigation/core/replay/MapboxReplayer.kt	`68.11% <100.00%> (-2.90%)`	⬇️
.../navigation/core/replay/route/ReplayRouteMapper.kt	`72.05% <100.00%> (ø)`
...ion/core/trip/session/TripSessionLocationEngine.kt	`86.11% <0.00%> (-5.56%)`	⬇️
... and 2 more

VysotskiVadim

@kmadsen , lazy initialisation is a great idea for replay events! It happened to me that I wasn't able to simulate 10Hz location updates for Gdansk-Lisbon route, app crashed with OOM

VysotskiVadim · 2022-11-23T09:30:08Z

...ion-core/src/main/java/com/mapbox/navigation/core/replay/route/ReplayPolylineDecodeStream.kt

+    override fun next(): Point {
+        var result = 1
+        var shift = 0
+        var temp: Int
+        do {
+            temp = encodedPath[index++].code - 63 - 1
+            result += temp shl shift
+            shift += 5
+        } while (temp >= 0x1f)
+        lat += if (result and 1 != 0) (result shr 1).inv() else result shr 1
+
+        result = 1
+        shift = 0
+        do {
+            temp = encodedPath[index++].code - 63 - 1
+            result += temp shl shift
+            shift += 5
+        } while (temp >= 0x1f)
+        lng += if (result and 1 != 0) (result shr 1).inv() else result shr 1
+
+        return Point.fromLngLat(lng / factor, lat / factor).also { next ->
+            current = next
+        }
+    }


RouteLine decodes full route anyway and put it into cache. Maybe instead of decoding geometry by small pieces you can get full decoded from the cache? Or in case route simulation was launched earlier than route line, you can decode full route on background thread and put it into the cache so that route line will reuse decoded geometry later.
What do you think about it?

I think there are a few options. Another option is putting the geometry string into a file (also can be a memory cache). We can decode with an InputStream mapbox/mapbox-java#1518.

Right now this ReplayPolylineDecodeStream is internal and considered temporary. I still want to decode it as a Iterable stream because that makes the ReplayRouteSession usage simpler. But i'm okay if geometry is stored in a file, cache, or memory.

looks good enough for the first iteration 👍

VysotskiVadim · 2022-11-23T09:57:32Z

libnavigation-core/src/main/java/com/mapbox/navigation/core/replay/route/ReplayRouteMapper.kt

+    /**
+     * Take a list of [Point] and map it to events that can be replayed by the [MapboxReplayer].
+     *
+     * @param points containing location coordinates to be replayed.
+     * @return [ReplayEventBase] [List]
+     */
+    fun mapPointList(points: List<Point>): List<ReplayEventBase> {
+        return replayRouteDriver.drivePointList(options, points)


I like current implementation 👍
But I think the same result could be achieved easier using Sequence

Suggested change

/**

* Take a list of [Point] and map it to events that can be replayed by the [MapboxReplayer].

*

* @param points containing location coordinates to be replayed.

* @return [ReplayEventBase] [List]

*/

fun mapPointList(points: List<Point>): List<ReplayEventBase> {

return replayRouteDriver.drivePointList(options, points)

/**

* Simulate a driver navigating a route

*

* @param geometry is a [DirectionRoute]

* @return [ReplayEventBase] [Sequence]

*/

suspend fun mapPointList(route: DirectionRoute): Sequence<ReplayEventBase> {

val decodeGeometry = withContext(Dispatchers.Default) {

route.completeGeometryToPoints()

}

return replayRouteDriver.drivePointList(options, points)

And if you replace lists by sequences in replayRouteDriver.drivePointList and all algorithms under the hood, every element of the sequence will be calculated on demand, i.e. lazily.

What do you think about it?

All of the functions in the ReplayRouteMapper return a List. And the replayer still takes events as a list
fun pushEvents(events: List<ReplayEventBase>)

So I don't think it is easier to convert back and forth at this time.

The algorithms under the covers are still using lists, because they dedup locations on the route and they estimate the route curvatures. I agree that iterable sequence can be better, but in this case we're also sharing algorithms and moving code back and forth between mapbox java https://github.com/mapbox/mapbox-java/blob/1b867fe8e68b299f37e5591c8431df89881ca9c2/services-geojson/src/main/java/com/mapbox/geojson/utils/PolylineUtils.java#L39. Some of the decisions were based on the tools that exist in mapbox-java

It will be a big change to turn everything into a sequence

Considering that we are trying to make replay "streamable", I also agree Sequence/Iterable/InputStream is where we should be going. But the ReplayRouteMapper is going to continue to be a List creator.

Take a look at ReplayRouteDriver, that is what we want to work on making into a Sequence. And then the mapper will convert the sequence into a List to maintain backwards compatibility.

I've been thinking about this too, so am just sharing the thoughts on this.

Considering that we are trying to make replay "streamable", I also agree Sequence/Iterable/InputStream is where we should be going. But the ReplayRouteMapper is going to continue to be a List creator.

Take a look at ReplayRouteDriver, that is what we want to work on making into a Sequence. And then the mapper will convert the sequence into a List to maintain backwards compatibility.

I agree, we have to leave current ReplayRouteMapper as is with List just because we can't break this API. What if we introduce new API with Sequence in parallel by copy pasting ReplayRouteMapper and replacing lists by sequences?

That is essentially the approach I was taking here #6582

Instead of a Sequence it's an Iterator. I'm not opposed to introducing it, but it still requires bug fixing

public interface ReplayEventStream extends Closeable, Iterator<ReplayEventBase>

VysotskiVadim · 2022-11-23T10:01:35Z

libnavigation-core/src/main/java/com/mapbox/navigation/core/replay/route/ReplayRouteSession.kt

+ * The replay session will be enabled when [MapboxNavigation] is attached.
+ */
+@ExperimentalPreviewMapboxNavigationAPI
+class ReplayRouteSession : MapboxNavigationObserver {


We have users who still create MapboxNavigaiton using constructor. Will they be able to use this lazy mechanism without migrating to MapboxNavigationApp?

Yeah. The MapboxNavigationObserver can be used without MapboxNavigationApp. The interface exposes the functions.

interface MapboxNavigationOberver { fun onAttached(mapboxNavigation: MapboxNavigation) fun onDetached(mapboxNavigation: MapboxNavigation) }

This means you can use ReplayRouteSession in this way. Call start and stop replay whenever you have an instance of MapboxNavigation. But this is NOT RECOMMENDED because we want to share this object with android auto.

val replayRouteSession = ReplayRouteSession() fun startReplay() { replayRouteSession.onAttached(mapboxNavigation)( } fun stopReplay() { replayRouteSession.onDetached(mapboxNavigation)( }

VysotskiVadim

I tested using the route Gdańsk(Poland)-Ho Chi Minh City(Vietnam) with 10Hz location update. On main branch the test app crashes with OOM. But with new optimisation simulation works fine, java heap doesn't grow more the 0.2 Gb 👍

Well done! 💪
Let's merge and iterate improving this solution 🚀
Improvements I recommend:

Try using Sequience, that would allow driver not to stop every X km because we don't process routes by chunks
Try reusing existing decoded route geometry from cache

kmadsen mentioned this pull request Nov 22, 2022

Add memory efficient polyline decoder mapbox/mapbox-java#1518

Open

kmadsen marked this pull request as ready for review November 22, 2022 01:01

kmadsen requested a review from a team as a code owner November 22, 2022 01:01

kmadsen force-pushed the km-NAVAND-728-memory-efficient-replay branch 3 times, most recently from 427788f to 99a6929 Compare November 23, 2022 01:30

Make a memory efficient replay

f1f6cca

kmadsen force-pushed the km-NAVAND-728-memory-efficient-replay branch from 99a6929 to f1f6cca Compare November 23, 2022 01:43

VysotskiVadim reviewed Nov 23, 2022

View reviewed changes

fix lint

263c216

VysotskiVadim self-requested a review November 24, 2022 10:55

VysotskiVadim approved these changes Nov 24, 2022

View reviewed changes

kmadsen merged commit 6cf020a into main Nov 28, 2022

kmadsen deleted the km-NAVAND-728-memory-efficient-replay branch November 28, 2022 16:38

This was referenced Nov 30, 2022

Add ReplayEventStream for replaying events while using constant memory #6582

Closed

Fix ReplayRouteSession route state #6675

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory efficient replay route #6636

Memory efficient replay route #6636

kmadsen commented Nov 19, 2022 •

edited

Loading

codecov bot commented Nov 23, 2022 •

edited

Loading

VysotskiVadim left a comment •

edited

Loading

VysotskiVadim Nov 23, 2022

kmadsen Nov 23, 2022 •

edited

Loading

kmadsen Nov 23, 2022 •

edited

Loading

VysotskiVadim Nov 24, 2022

VysotskiVadim Nov 23, 2022

kmadsen Nov 23, 2022

kmadsen Nov 23, 2022 •

edited

Loading

kmadsen Nov 23, 2022

VysotskiVadim Nov 24, 2022

kmadsen Nov 28, 2022 •

edited

Loading

VysotskiVadim Nov 23, 2022

kmadsen Nov 23, 2022 •

edited

Loading

VysotskiVadim Nov 24, 2022

VysotskiVadim left a comment •

edited

Loading

Memory efficient replay route #6636

Memory efficient replay route #6636

Conversation

kmadsen commented Nov 19, 2022 • edited Loading

Description

Api explanation

Known issue

What kind of memory improvement to expect

codecov bot commented Nov 23, 2022 • edited Loading

Codecov Report

VysotskiVadim left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmadsen Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

kmadsen Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmadsen Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmadsen Nov 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmadsen Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VysotskiVadim left a comment • edited Loading

Choose a reason for hiding this comment

kmadsen commented Nov 19, 2022 •

edited

Loading

codecov bot commented Nov 23, 2022 •

edited

Loading

VysotskiVadim left a comment •

edited

Loading

kmadsen Nov 23, 2022 •

edited

Loading

kmadsen Nov 23, 2022 •

edited

Loading

kmadsen Nov 23, 2022 •

edited

Loading

kmadsen Nov 28, 2022 •

edited

Loading

kmadsen Nov 23, 2022 •

edited

Loading

VysotskiVadim left a comment •

edited

Loading