Update to Android 10

[ploink]

I recently upgraded to Android 10 and GSMbackend does not work reliably. When I disable/enable the backend it may work briefly and then stop again.
Part of it may be related to this issue with microg but I think there is more to it. Even with android 9, two of my apps that only use coarse location rarely get a location while Osmand does most of the time.

So I forked the project and tried to update it. At first it would not build with Android studio 3.5.3, but I managed to fix that by updating gradle and some dependencies. I managed to get it up to SDK28 (Android 9). Then I converted to AndroidX (which is required for SDK29). The problem with SDK29 (Android 10) is that method getNeighboringCellInfo() is deprecated and no longer available. Even when I target SDK28, the method does not return any data on the device. Code changes are needed to fix that.

I published two apk's from the updated source if anyone wants to try. For me the debug version actually works much better on A10, about the same as it did on A9. It would be interesting to figure out why.
Any comments appreciated.

[n76]

It is highly unlikely that I'll be maintaining this project going forward. Would you like to take it over?

I am not sure how that is done on GitHub but I am sure we could figure it out.

[ArchangeGabriel]

You can transfer ownership of a repository.

[ploink]

Thank you for the honor but I doubt I am worthy. I have only just started to play with Android app development and this is the second app I even managed to compile. I don't really have the capacity to manage such a project on my own. But I will keep tinkering for a while and try to get it working properly on Android 10.

Note that you can archive this project any time if you wish. If you do than I will detach my fork and rename it so it is no longer connected to your "org.fitchfamily" domain.

Maybe you could donate this project to microg if they want it, then multiple people can work on it. They already have 4 backends and then I can just offer my contribution there. This would be the best solution I think.

[baldrianbandit]

@ploink I would like to test your updated build, but I cannot see any new releases in https://github.com/ploink/Local-GSM-Backend/releases

Thank you in advance

[ploink]

@baldrianbandit Sorry about that. I just uploaded a new release. This one should work.
I had removed the previous one because it was not quite ready yet.

[breversa]

@ploink I'm testing v1.4.23-ploink right now. It works awesomely fine once set up following your instructions. Great work ! :D

[baldrianbandit]

Unfortunately, my attempts did not lead to a working network location, yet. I
-uninstalled 1.4.22 via adb
-removed all findings of org.fitchfamily from /data
-rebooted and cleared ART/cache
-installed 1.4.23-ploink via adb
-granted all time location access to 1.4.23 and removed battery management on it

Thereby
-the API29 warnings disappeared from system log
-some new warnings appeared:
GnssLocationProvider: sendExtraCommand: unknown command force_xtra_injection
NlpLocationProvider: onSetRequest: ProviderRequest[OFF] by WorkSource{}
NlpLocationProvider: using autoUpdate=false autoTime=9223372036854775807

which show up repeatedly while Osmand is active. It does not receive a network location.

[ploink]

@baldrianbandit
The first line tells me your GPS does not support injecting Xtra data on command, which is not really a problem. I guess it is Osmand attempting to speed up GPS signal acquisition, but the firmware usually does this in the background whenever it thinks it is needed.

I am not sure about the other lines, but you may want to look for "ProviderRequest[ON interval...." lines and "using autoUpdate=true autoTime=".
These are initiated by Android to schedule location updates. The daily updates are for providing lastknown locations when apps requesting a location are in the background. They need background location access to get fresh data in Android 10 when they are in the background. Faster updates are from apps in foreground (Osmand asks for interval=0 but my phone limits that to 2500ms).

I have added a debug apk to the releases. This one should give you a lot more info if you filter/search logcat for "gsmloc". You can simply install it as an update over the release version. Disable, then re-enable the backend in microg's "configure location backends" to get it going.

You could use my app https://github.com/ploink/Hellocation for testing if you like.

[baldrianbandit]

Thanks a lot. 1.4.23-debug is much more chatty. Also, Hellocation is very appreciated - it's good to clearly differentiate between gps and network location.

But, in the end, Hellocation does not get any network location updates using 1.4.23-debug as a provider. Here is what gsmloc logs on my LineageOS 17.1 device:

gsmloc_TelephonyHelper: New timestamp: 78113931216
gsmloc_TelephonyHelper: CellInfo: WCDMA MCC=262 MNC=-1 CID=13183137 LAC=15 dBm=-67
gsmloc_service: onCellInfo() callback!
gsmloc_service: update()
gsmloc_service: onCellInfoChanged(): [CellInfoWcdma:{mRegistered=YES mTimeStamp=141310144386ns mCellConnectionStatus=2147483647 CellIdentityWcdma:{ mLac=15 mCid=13183137 mPsc=204 mUarfcn=10564 mMcc=262 mMnc=null mAlphaLong= mAlphaShort=} CellSignalStrengthWcdma: ss=-65 ber=99 rscp=2147483647 ecno=2147483647 level=4}, CellInfoWcdma:{mRegistered=NO mTimeStamp=141310144386ns mCellConnectionStatus=2147483647 CellIdentityWcdma:{ mLac=2147483647 mCid=2147483647 mPsc=251 mUarfcn=10564 mMcc=null mMnc=null mAlphaLong= mAlphaShort=} CellSignalStrengthWcdma: ss=-79 ber=99 rscp=2147483647 ecno=2147483647 level=3}]

[ploink]

@baldrianbandit

WCDMA MCC=262 MNC=-1 CID=13183137 LAC=15 dBm=-67
mMcc=262 mMnc=null

Looks like you have been bitten by the 0-prefix MNC bug in Android 9 and possibly 10.
microg/IchnaeaNlpBackend#50
Are you using Android 9 and can you confirm your MNC starts with a 0?
https://www.mcc-mnc.com/

I can fix that but it will take some time. Gsm-backend uses UnifiedNlpApi but has its own code for fetching cell info.

[baldrianbandit]

Yeah, but isn't the 0-prefix MNC bug is related to IchnaeaBackend, only? @domkrm even published a fixed build (see below in the linked issue). But I prefer a network location service that doesn't query my location info at any online database.

The point is, reverting back to LocalGSMBackend 1.4.22 makes my network location work again. But I cannot find any hint in logcat that shows me why it isn't working when using 1.4.23-ploink. You are right, the MNC=-1 looks suspicious. But why should 1.4.22 not suffer from the same behaviour? (Although I cannot see, because it's less chatty than your build)

[ploink]

Yeah, but isn't the 0-prefix MNC bug is related to IchnaeaBackend, only?

It is a bug in Android. The Android getCellIdentity() method returns "mMcc=null" so the value is simply not available. It would affect all apps that try this.

What domkrm's fix does, it calls getNetworkOperator() when getCellIdentity() does not return a valid MNC. I am not sure though how that deals with multi-sim situations.

If the old v1.4.22 does give the MNC, then that is interesting. It uses the now deprecated getCellLocation() and getNeighboringCellInfo(). These may not have the MNC bug. Problem is, they are no longer available when compiling for Android 10.

I would have to look into it further and see what is the best way to work around the bug and still support multi-sim properly.

[ploink]

@baldrianbandit You may try v1.4.24-ploink. It includes the workaround.

[baldrianbandit]

Wow @ploink. Hellocation instantly get's a location fix via network using 1.4.24-ploink. Thanks a lot.

gsmloc_service: [...] mMcc=null mMnc=null [...]
gsmloc_TelephonyHelper: CellInfo: WCDMA MCC=262 MNC=-1
gsmloc_TelephonyHelper: Fixed MNC=2

Is that part of the workaround -- do you assign a fixed MNC?

[ploink]

I am glad it works. If you look at TelephonyHelper.java there is this bit of code:

            if (mcc <= 0) continue;

            // Uncomment to simulate 0-prefix MNC bug
            //if (DEBUG) mnc = -1;

            Log.i(TAG, "CellInfo: " + tech + " MCC=" + mcc + " MNC="
                    + mnc + " CID=" + cid + " LAC=" + lac + " dBm=" + Math.round(dBm));
            if (cid == CellInfo.UNAVAILABLE || lac == CellInfo.UNAVAILABLE
                    || mcc < 200 || mcc == 999) continue;


            // Workaround for 0-prefix MNC bug
            if (mnc < 0 && inputCellInfo.isRegistered()) {
                mnc = toInteger(tm.getNetworkOperator().substring(3));
                Log.w(TAG, "Fixed MNC=" + mnc);
            }

In plain english: If MCC, CID & LAC are good and MNC is missing, then use your currently registered network operator MNC code.

The fix is not perfect. You only use data from the registered cell so you never get an average of multiple cell tower locations. This is because you cannot be certain the non-registered cells have the same MNC. But this is not much of a problem as most phones rarely get that info for non-registered cell towers.

EDIT: According to This answer, you only get neighboring(non-registered) cell info when you are on GSM. I just tried forcing my phone to GSM and indeed, I now pick up 6 cell towers in my area with valid mcc/mnc/lac/cid and get a much more accurate location.

It gets more complicated when you use multiple SIM cards and are registered to more than one cell tower at the same time and they are from different providers. Then you may get the wrong MNC code.

[baldrianbandit]

My network based location when using gsmloc has always been a little blurred, as if the service knew the current cell, only. I find that okay since it becomes accurate as soon as I get an additional GPS fix.

As you mentioned the now deprecated getNeighboringCellInfo() - I think it never worked on my device, thus it also seems to suffer from the 0-prefix bug.

In TelephonyHelper the MNC is assign by

mnc = toInteger(id.getMncString());

If I had a MNC of '02' (str), it would become '2' (int) and thus, it would match my MNC in a db query no more. Do you think the root of the whole issue is Android handling MNCs as integers (or having 0-prefixed numbers at all)?

[ploink]

The database is in integers. With your phone/rom "id.getMncString()" always returns null.Theoretically MNC 02 could be different from 002, but that confusion is avoided when assigning MNC's. I forgot where I read that, but for example this list also uses integers.

[Chris56]

Sorry for being so ignorant, but having dectected that the local cell based location on LOS 17.1 with microg didn't work, I came acoss this thread ..
A few questions:
a) For testing: is there a Hellocation apk to use for installation?
b) When I use your https://github.com/ploink/Local-GSM-Backend/releases/download/v1.4.25-ploink/GsmBackend-1.4.25-ploink-release.apk
I will first disable and the uninstall the f-droid Local-GSM-Backend. Afterward I install your apk and enable it.
Anything else I have to do? Like I read: "...removed battery management ..." Should I do that and if yes: how?
Sorry abount my ignorance!
Chris

[ploink]

@Chris56
a) Yes there is one release https://github.com/ploink/Hellocation/releases

I will first disable and the uninstall the f-droid Local-GSM-Backend. Afterward I install your apk and enable it.

Correct, but also in microG setting:

• Press the text "GSM Location module service" (next to the switch)
• Then press "configure"
• Then from the menu, select settings and there you select the area's for which you want to download the database.
• Then you select Database from the menu and you download/update the database.

You do not need to disable battery optimization for GSM Location service. It should run just fine with a recent MicroG version.

[starbrights]

I am using (or try to use) this with LOS17.1 and microG. Downloaded from FDroid (1.4.23).
I am using SatStat and OsmAnd to see the results. It seems not to work, or only sometimes.
In log I see this errors:
"Allowing org.fitchfamily.android.gmslocation fine cecause it doesn't target API 29 yet. Please fix this app. Called from getCellLocation"
"Allowing org.fitchfamily.android.gmslocation fine cecause it doesn't target API 29 yet. Please fix this app. Called from getCellInfo"
What exactly does it mean?

The hellocation app does only show No last known locatipon (for any type). May I ask what is difference between passive and network?
After activating Interval there is just: Started location update... (only for GPS I get a position).

After once a GPS position is found, it repeats that also in LastKnow when selecting "passive". Is this the expected behaviour?

The appleWifi Backend on the other hand returns just one! passive position after some seconds (although interval is set to 0".

[ploink]

@starbrights
Part of your problem may be due to microg/GmsCore#1196
With current gmscore, interval is minimum 30s and no location is requested from the backend until the timer expires.

API 29 = Android 10.
You may try my fork, it is updated to Android 10 and adds 4G and 5G capability.
(uninstall current version first, it is signed with a different key)
https://github.com/ploink/Local-GSM-Backend/releases/tag/v1.4.25-ploink
Or the one from deveee on gitlab who merged some of my commits:
https://gitlab.com/deveee/Local-GSM-Backend

May I ask what is difference between passive and network?
https://developer.android.com/reference/android/location/LocationManager#PASSIVE_PROVIDER
But note that this is implemented in Gapps and MicroG might behave slightly differently.

After once a GPS position is found, it repeats that also in LastKnow when selecting "passive". Is this the expected behaviour?
Yes, Because passive provider can get a location from other providers, gps or network, whichever is available.

[n76]

I have handed off this project to @Devee. The new official repository for this work is a now at https://gitlab.com/deveee/Local-GSM-Backend.

The current f-droid version of this seems to be built based on the version at the new maintainer's repository so if you are using that version there should be no issue with the apk signature.

If this is still an issue, please open it on the new repository with the new maintainer.

[starbrights]

I can confirm the 1.4.25-ploink works! While I think latest FDroid vesion not.
Thank you! I also found that 5s on / 30s off intervall, in fact it is my report of issue.

For me it is hard to see is it microG, backend, app, OS or my stupidness that it fails. Lot of things can fail and I'm not an expert.

Imho it would be good to bring back the energy saving location service as in Android 9. At least as an option in LOS.

[deveee]

@starbrights I have to fix one crash that happens on old android version after updating the project to android 10 and then I think that it can be updated on fdroid.

[starbrights]

Thank you and of course @ploink for all that.

[jwrdegoede]

@starbrights wrote:

I also found that 5s on / 30s off intervall, in fact it is my report of issue.

Copying my reply to: microg/GmsCore#1196 about this here to keep everyone in the loop:

I'm seeing similar network location timing issues on a lineage-17.1 + latest micro-G config. I've run several tests and the amount of time between updates (and the amount of time when only a "stale" location is available) seems to depend on which Nlp providers are used:

• With GSM location service the network location is ok for 5 seconds and then shown as stale in SatStats for about 55 seconds for me.
• Networklocation works with 5 seconds ok + 10 seconds stale data when downgrading mozilla NLP (IchnaeaNlpBackend) to version 1.3.3 (and using this as the sole NLP provider)
• Where as with mozilla NLP 1.5.0 I only see a proper up2date network location once every minute / couple of minutes and when starting an app inside the cooldown window it seems no data is provided instead of the last location with a flag that it is old-data, breaking many apps
• The mozilla NLP 1.5.0 issue is likely caused by: microg/IchnaeaNlpBackend@174e736 . Maybe in combination with GmsCore including the UnifiedNlp v2.0-alpha# versions
• Networklocation works fine when using Apple Wifi NLP as sole provider

If you have no objection against using the Apple Wifi webAPIs (and thus sending some data to Apple) then using the Apple Wifi NLP provider from F-Droid is a good workaround for the timings issues. At least this seems to work well for me.

[jwrdegoede]

A quick update on this, I just retried this on a fresh install and for the apple backend to work properly I also need to disable the "throttle wifi scanning" option under developer options and then disable + re-enable the apple wifi backend in the microg location settings.

Without this the apple backend was behaving quite similar to the mozilla NLP (IchnaeaNlpBackend) version 1.3.3 case, where Networklocation works with 5 seconds ok + 10 seconds stale data. Unfortunately disabling "throttle wifi scanning" does not help for the 1.3.3 mozilla NLP backend.