make pygpsclient compatible with gpsd and ubxtool

[schaefer01]

PyGPSClient Feature Request Template

I'd like the graphical ability to view satellites in parallel with commanding detailed configuration by gpsd and ubxtool

Only one device can access the ublox port on a device and the combination gpsd and ubxtool provide more command-ability than pygpsclient but gpsd and ubxtool are command line and not graphical

A clear and concise description of what the problem is e.g. I'd like to be able to do this [...]

It would be nice to have pygpsclient request info from gpsd rather than from the device

Describe alternatives you've considered

1. stop the gpds daemon, then run pygpsclient

2. exit pygpsclient then start the gpsd daemon

Would you be willing to contribute a test device?

If the request relates to a specific u-blox device that is not currently supported, would you be
willing to contribute a device to the project for testing purposes?

[semuadmin]

Hi @schaefer01,

Sorry, but that would involve a significant amount of (unpaid) work, both to implement and test and then to keep in line with any subsequent gpsd or ubxtool API updates. I simply don't have the time myself, but happy to consider contributions.

Out of interest, what specific satellite information would you want to see displayed? There may be other ways to achieve this natively.

I'll move this to the pygnssutils Discussion Forum for others to follow up on.

[schaefer01]

hi, semuadmin,
Immediately after I posted this I verified that I haven't set up the gpsd .conf file,
the gpsd documentation (I didn't RTFM) seems to show that gpsd can put data out on a socket,.
I'm going to try that next. I apologize for spinning you up.

[semuadmin]

hi, semuadmin, Immediately after I posted this I verified that I haven't set up the gpsd .conf file, the gpsd documentation (I didn't RTFM) seems to show that gpsd can put data out on a socket,. I'm going to try that next. I apologize for spinning you up.

No worries - yes, gpsd typically outputs JSON data to a TCP socket on port 2947, and this is how most users consume the data. Consuming a third party data stream from a TCP socket is not in itself particularly difficult (bit trickier if encrypted) - this is essentially what PyGPSClient already does with its NTRIP and SPARTN client facilities - but parsing the JSON and displaying the data would involve a lot more work. It is, however, perfectly possible.

But, as I say, if you can clarify which specific additional satellite information you would wish to see displayed, it may be possible to obtain this from existing UBX sources and display in a new user-selectable 'widget', which is a relatively straightforward change.

[schaefer01]

Follow up question, since gpsd and ubxtool are open source, and ubxtool decodes setting ubx commands that are not in pygpsclient, and these commands are not availaible in pygpsclient, do you see the possibility of the maintainer(s) of pygpsclient adding a few additional settable ublox-related parameters? In particular I'm looking for setting the ubx firmware and enabling ubx raw messages. I ask because these parameters are needed for configuring ublox for data collection to pass to a rinex tool where my science group at MIT Haystack is interested in pulling out TEC values from the rinex conversion. Setting firmware allows for a configuration that survives power cycling, and raw messages passes TEC related information. A no from you is ok, just asking.

[semuadmin]

Follow up question, since gpsd and ubxtool are open source, and ubxtool decodes setting ubx commands that are not in pygpsclient, and these commands are not availaible in pygpsclient,

To the best of my knowledge, PyGPSClient (via its underlying UBX parser pyubx2) supports all public domain UBX SET commands (from Generation 6 through Generation 10 protocols), and is fully capable of 'decoding' binary streams containing SET UBX messages provided Msg Mode is set to 'SET' or 'SETPOLL' as opposed to 'GET' *:

* the necessity for the distinction arises from the fact that ublox uses the same msg class and id for GET and SET messages, but they have different payload definitions.

Any messages that might be missing can easily be added provided they're documented in public domain ublox Interface Specifications.

do you see the possibility of the maintainer(s) of pygpsclient adding a few additional settable ublox-related parameters? In particular I'm looking for setting the ubx firmware and enabling ubx raw messages.

If, by 'ubx raw messages', you're referring to the UBX-RXM-RAWX and UBX-RXM-SFRBX message types, these are already supported by PyGPSClient/pyubx2 and can be enabled (where supported by the receiver) via the UBX Config Dialog.

• UBX-RXM-RAWX contains the information needed to be able to generate a RINEX 3 multi-GNSS observation file (see ftp://ftp.igs.org/pub/data/format/).
• UBX-RXM-SFRBX contains a complete subframe of broadcast navigation data decoded from a single signal. The number of data words reported in each message depends on the nature of the signal.

e.g. here's a RXM-RAWX message from an F9N receiver parsed by PyGPSClient:

<UBX(RXM-RAWX, rcvTow=223793.0, week=2327, leapS=18, numMeas=23, leapSec=1, clkReset=0, reserved1=9034753, prMes_01=21431659.961167824, cpMes_01=112624126.09217028, doMes_01=366.0107421875, gnssId_01=GPS, svId_01=11, sigId_01=0, freqId_01=0, locktime_01=64500, cno_01=43, prStd_01=4, cpStd_01=1, doStd_01=6, prValid_01=1, cpValid_01=1, halfCyc_01=1, subHalfCyc_01=0, reserved3_01=0, prMes_02=23388266.420824006, cpMes_02=121788836.88164197, doMes_02=-1343.677001953125, gnssId_02=BeiDou, svId_02=11, sigId_02=0, freqId_02=0, locktime_02=64500, cno_02=45, prStd_02=3, cpStd_02=1, doStd_02=6, prValid_02=1, cpValid_02=1, halfCyc_02=1, subHalfCyc_02=0, reserved3_02=0, prMes_03=21944027.967587065, cpMes_03=114268322.28770317, doMes_03=-6.217041492462158, gnssId_03=BeiDou, svId_03=21, sigId_03=0, freqId_03=0, locktime_03=64500, cno_03=43, prStd_03=3, cpStd_03=1, doStd_03=6, prValid_03=1, cpValid_03=1, halfCyc_03=1, subHalfCyc_03=1, reserved3_03=0, prMes_04=24290964.67247113, cpMes_04=126489440.82625493, doMes_04=2302.086181640625, gnssId_04=BeiDou, svId_04=42, sigId_04=0, freqId_04=0, locktime_04=64500, cno_04=48, prStd_04=3, cpStd_04=1, doStd_04=5, prValid_04=1, cpValid_04=1, halfCyc_04=1, subHalfCyc_04=0, reserved3_04=0, prMes_05=22660122.874939907, cpMes_05=117997202.83262888, doMes_05=1363.71875, gnssId_05=BeiDou, svId_05=50, sigId_05=0, freqId_05=0, locktime_05=64500, cno_05=41, prStd_05=4, cpStd_05=2, doStd_05=7, prValid_05=1, cpValid_05=1, halfCyc_05=1, subHalfCyc_05=1, reserved3_05=0, prMes_06=24132893.363859024, cpMes_06=125666285.33264719, doMes_06=-2736.27587890625, gnssId_06=BeiDou, svId_06=34, sigId_06=0, freqId_06=0, locktime_06=64500, cno_06=45, prStd_06=3, cpStd_06=1, doStd_06=6, prValid_06=1, cpValid_06=1, halfCyc_06=1, subHalfCyc_06=1, reserved3_06=0, prMes_07=20505357.148789547, cpMes_07=109766763.53250594, doMes_07=2015.007568359375, gnssId_07=GLONASS, svId_07=3, sigId_07=0, freqId_07=12, locktime_07=64500, cno_07=42, prStd_07=5, cpStd_07=2, doStd_07=6, prValid_07=1, cpValid_07=1, halfCyc_07=1, subHalfCyc_07=0, reserved3_07=0, prMes_08=20222732.420179326, cpMes_08=107950329.93927452, doMes_08=-1145.3233642578125, gnssId_08=GLONASS, svId_08=18, sigId_08=0, freqId_08=4, locktime_08=64500, cno_08=46, prStd_08=5, cpStd_08=1, doStd_08=5, prValid_08=1, cpValid_08=1, halfCyc_08=1, subHalfCyc_08=0, reserved3_08=0, prMes_09=21889633.295237724, cpMes_09=117135849.04657419, doMes_09=-4390.41845703125, gnssId_09=GLONASS, svId_09=17, sigId_09=0, freqId_09=11, locktime_09=0, cno_09=25, prStd_09=9, cpStd_09=15, doStd_09=10, prValid_09=1, cpValid_09=0, halfCyc_09=0, subHalfCyc_09=0, reserved3_09=0, prMes_10=21686508.780911602, cpMes_10=116008198.02799152, doMes_10=2402.61279296875, gnssId_10=GLONASS, svId_10=19, sigId_10=0, freqId_10=10, locktime_10=0, cno_10=26, prStd_10=9, cpStd_10=15, doStd_10=10, prValid_10=1, cpValid_10=0, halfCyc_10=0, subHalfCyc_10=0, reserved3_10=0, prMes_11=22018183.361059923, cpMes_11=117658468.23311494, doMes_11=-1140.9718017578125, gnssId_11=GLONASS, svId_11=11, sigId_11=0, freqId_11=7, locktime_11=64500, cno_11=28, prStd_11=7, cpStd_11=7, doStd_11=8, prValid_11=1, cpValid_11=1, halfCyc_11=1, subHalfCyc_11=1, reserved3_11=0, prMes_12=22743120.940194048, cpMes_12=121575015.44618322, doMes_12=-3792.70654296875, gnssId_12=GLONASS, svId_12=1, sigId_12=0, freqId_12=8, locktime_12=0, cno_12=18, prStd_12=10, cpStd_12=15, doStd_12=12, prValid_12=1, cpValid_12=0, halfCyc_12=0, subHalfCyc_12=0, reserved3_12=0, prMes_13=23858352.861107133, cpMes_13=125376443.5165817, doMes_13=-1376.6234130859375, gnssId_13=Galileo, svId_13=8, sigId_13=0, freqId_13=0, locktime_13=14000, cno_13=28, prStd_13=8, cpStd_13=6, doStd_13=8, prValid_13=1, cpValid_13=1, halfCyc_13=1, subHalfCyc_13=0, reserved3_13=0, prMes_14=24214677.327190127, cpMes_14=127248980.04656897, doMes_14=1358.4405517578125, gnssId_14=Galileo, svId_14=7, sigId_14=0, freqId_14=0, locktime_14=64500, cno_14=48, prStd_14=3, cpStd_14=1, doStd_14=5, prValid_14=1, cpValid_14=1, halfCyc_14=1, subHalfCyc_14=0, reserved3_14=0, prMes_15=21565176.16468655, cpMes_15=113325769.79889342, doMes_15=-2675.572509765625, gnssId_15=GPS, svId_15=6, sigId_15=0, freqId_15=0, locktime_15=64500, cno_15=33, prStd_15=6, cpStd_15=5, doStd_15=8, prValid_15=1, cpValid_15=1, halfCyc_15=1, subHalfCyc_15=1, reserved3_15=0, prMes_16=24731761.429656383, cpMes_16=129966287.52342677, doMes_16=884.7816772460938, gnssId_16=GPS, svId_16=29, sigId_16=0, freqId_16=0, locktime_16=43240, cno_16=28, prStd_16=7, cpStd_16=7, doStd_16=8, prValid_16=1, cpValid_16=1, halfCyc_16=1, subHalfCyc_16=0, reserved3_16=0, prMes_17=40842571.425258644, cpMes_17=212677986.41371614, doMes_17=-1434.8870849609375, gnssId_17=BeiDou, svId_17=16, sigId_17=0, freqId_17=0, locktime_17=0, cno_17=24, prStd_17=8, cpStd_17=15, doStd_17=10, prValid_17=1, cpValid_17=0, halfCyc_17=0, subHalfCyc_17=0, reserved3_17=0, prMes_18=19675402.925460763, cpMes_18=104991719.66134515, doMes_18=-1496.8182373046875, gnssId_18=GLONASS, svId_18=2, sigId_18=0, freqId_18=3, locktime_18=0, cno_18=19, prStd_18=10, cpStd_18=15, doStd_18=12, prValid_18=1, cpValid_18=0, halfCyc_18=0, subHalfCyc_18=0, reserved3_18=0, prMes_19=22155358.561108023, cpMes_19=118349947.90421374, doMes_19=2893.86279296875, gnssId_19=GLONASS, svId_19=12, sigId_19=0, freqId_19=6, locktime_19=0, cno_19=18, prStd_19=10, cpStd_19=15, doStd_19=12, prValid_19=1, cpValid_19=0, halfCyc_19=0, subHalfCyc_19=0, reserved3_19=0, prMes_20=22314327.690778643, cpMes_20=117262596.46916032, doMes_20=1962.57275390625, gnssId_20=GPS, svId_20=7, sigId_20=0, freqId_20=0, locktime_20=41440, cno_20=28, prStd_20=6, cpStd_20=6, doStd_20=8, prValid_20=1, cpValid_20=1, halfCyc_20=1, subHalfCyc_20=1, reserved3_20=0, prMes_21=24994747.51350373, cpMes_21=131348310.08543116, doMes_21=-2783.71044921875, gnssId_21=Galileo, svId_21=13, sigId_21=0, freqId_21=0, locktime_21=64500, cno_21=27, prStd_21=7, cpStd_21=7, doStd_21=8, prValid_21=1, cpValid_21=1, halfCyc_21=1, subHalfCyc_21=0, reserved3_21=0, prMes_22=23955077.369382612, cpMes_22=125884769.58434573, doMes_22=94.6787109375, gnssId_22=Galileo, svId_22=26, sigId_22=0, freqId_22=0, locktime_22=64500, cno_22=35, prStd_22=5, cpStd_22=4, doStd_22=7, prValid_22=1, cpValid_22=1, halfCyc_22=1, subHalfCyc_22=0, reserved3_22=0, prMes_23=27440101.1398, cpMes_23=144198704.43711296, doMes_23=-3019.35888671875, gnssId_23=Galileo, svId_23=3, sigId_23=0, freqId_23=0, locktime_23=0, cno_23=22, prStd_23=9, cpStd_23=15, doStd_23=12, prValid_23=1, cpValid_23=0, halfCyc_23=0, subHalfCyc_23=0, reserved3_23=0)>

Note, however, that PyGPSClient/pyubx2 does not perform any kind of RINEX conversion of the RXM-RAWX data and it is not capable of collating complete NAV datasets from a time series of individual RXM-SFRBX subframes.

Firmware updates is unfortunately a bit of a sore point (see previous Discussions on this topic e.g. #104, ). It is possible to upgrade ublox firmware via a series of UBX commands, but the protocols and messages definitions are not publicly available from ublox. They can be requested under NDA, but ublox have expressly forbidden me to include the proprietary functionailty in any of my open source Python GNSS libraries. I'm aware that other developers take the view that, since the firmware upgrade protocols can be 'reverse engineered' via standard tooling (e.g. via packet sniffing), they are 'fair game' for open source software, but for now at least I'm being guided by ublox. This position may change at some point in the future.

I ask because these parameters are needed for configuring ublox for data collection to pass to a rinex tool where my science group at MIT Haystack is interested in pulling out TEC values from the rinex conversion. Setting firmware allows for a configuration that survives power cycling, and raw messages passes TEC related information. A no from you is ok, just asking.

You don't necessarily need to perform a firmware upgrade in order to persist a given UBX message cohort (indeed this would be something of a sledgehammer to crack a nut) - you can simply enable the UBX messages you want and then save this configuration to non-volatile memory (e.g. Battery-Backed RAM (BBR) or Flash). PyGPSClient fully supports this via the UBX Config Dialog. PyGPSClient also allows you to 'record' a given set of command inputs so you can 'replay' them at a later date.

[semuadmin]

Hi @schaefer01

Was the above helpful at all? If I understand your use case correctly, I think the UBX RXM-RAWX message type is what you're after (in which case PyGPSClient or pyubx2 can provide this data natively, without recourse to any gpsd or ubxtool integration), but happy to explore further.

[schaefer01]

HI,

I'm trying to be pragmatic with respect to my assignment as a research engineer
in assisting in development a new low cost gnss board for collecting TEC for science.
With assistance from hardware focused research engineers,I'm coding the first software on new
hardware, I'm coding device drivers but also proving the path from data collection to science-ready format.
The u-center and gpsd ubxtools do the heavy lifting, I can program and get non-graphical status
that the satellite "fix" is available so data collection should work.

the graphical interface pygpsclient provides is nice but not critical to my needs since I'd just be
using it to verify a "fix"

that said, once a year we get the opportunity to have an undergrad work on a summer project and next
year in the spring, if nothing else is higher priority, I'll request a student to provide an gpsd
interface to pygpsclient.

bob s

[semuadmin]

that said, once a year we get the opportunity to have an undergrad work on a summer project and next year in the spring, if nothing else is higher priority, I'll request a student to provide an gpsd interface to pygpsclient.

If you have no need for a GUI, then the pygnssutils library offers comparable CLI functionality.

Without wishing to labour this, I'd genuinely like to understand which specific gnss data attributes you feel you can get via gpsd that you can't get via PyGPSClient (or its underlying libraries pyubx2 and pygnssutils). I appreciate there may be presentational differences, but to the best of my knowledge there's nothing that gpsd can provide that PyGPSClient/pyubx2/pygnssutils can't also provide natively (provided the receiver is appropriately configured to output the requisite data). They're both accessing the same raw data stream from the receiver (NMEA, UBX or RTCM) - all gpsd is essentially doing is making that data available in JSON format via a TCP socket.

As I say, if you're looking to derive TEC from a RINEX conversion of the 'raw' data stream (a fairly common academic use case), then I suspect the UBX-RXM-RAWX data may well be your principal source, as it provides raw observational data for pseudorange, carrier phase and doppler shift. Most RTK-capable u-blox receivers (e.g. F9* series) can be configured to output this message type and this configuration can readily be saved to non-volatile memory without recourse to firmware upgrades.

I'd be more than happy to consider any contributions your team may wish to make, but I would hate for the work to be nugatory if in fact the need can be met via existing functionality (or minor enhancements thereof).

[schaefer01]

hi,
Efficiency of the effort-to-science pipeline.
No new code. No teaching scientists to write programs.
The scientist just invokes a script of command-line statements.
I don't need to debug programs.

[semuadmin]

Just to close this off...

Plenty of good reasons to use a well-established C CLI utility like gpsd - absolutely no argument there (I use it extensively myself).

What I was trying (unsuccessfully) to clarify in this discussion was why there would be a need to - in your own words - 'make pygpsclient compatible with gpsd and ubxtool' or 'provide a gpsd interface to pygpsclient'.

I'll close this now, but always happy to consider feature requests.

[schaefer01]

the need? the coolness of remote logging into a set of embedded linux boxes via xterms to retrieve graphical views, an xterm per remote box, each remote box systemd’d to collect data, where the local site providing a dashboard of remote statuses On Aug 14, 2024, at 9:19 AM, SEMU Admin ***@***.***> wrote: Just to close this off... Plenty of good reasons to use a well-established C CLI utility like gpsd - absolutely no argument there (I use it extensively myself). What I was trying (unsuccessfully) to clarify in this discussion was why there would be a need to - in your own words - 'make pygpsclient compatible with gpsd and ubxtool' or 'provide a gpsd interface to pygpsclient'. I'll close this now, but always happy to consider feature requests. — Reply to this email directly, view it on GitHub<#145 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMS2KEOKT63BUA5F3IZTVE3ZRNKNXAVCNFSM6AAAAABMCT4YUKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZTG4ZTSOI>. You are receiving this because you were mentioned.Message ID: ***@***.***>