gptp does not work well on NXP rt series platform #33747
Comments
What does the "NOT pass" mean here? Do you mean that the in 1020EVK, the for loop is not entered? The gptp subsystem is not board specific so this indicates some issue in mcux driver and/or NXP HAL. |
|
My interpretation was that the GPTP_PORT_END condition was not allowing the loop to be entered even one time, so it was getting optimized out by the compiler. Could it be that is a HAL issue then? We also have access to a couple of RT1050 EVKs, though they are in a different office. Would it be helpful to run the same test on a 1050? |
Yes, that would be useful information to know if there are issues in 1050 too. |
i.MX RT and Kinetis use the same HAL driver (fsl_enet.c).
Yes, please. @hakehuang can you have a look? |
|
@kevin137 do you follow the document in gptp sample document to set up the environment I find an issue on frdm_k64f and can reproduce it on mimxrt1060_evk, #29599 can you try the SDK ptp1588 example which can be download from kex.nxp.com, the log below shows that the board is ok. besides, do you add the board setting to mimxrt1020_evk, in gpt sample as below |
Ok. We should be able to do that. There are some logistical issues because of the Easter vacation, but we should be able to perform the test on a 1050-EVK early next week and post the results here. |
@kevin137 do you enable below config in mimxrt1020_evk? |
Yes, SDK ptp1588 example works as you mentioned for MIMXRT1020 and FRDM-k64f, although the PTP timestamp expected would be the same as in the timestamp in the master node (a Linux ptp4l implementation) as in Zephyr OS, right? Or why is the timestamp beginning at 0.0 seconds? Is the MIMXRT1020 or FRDM-k64f working as a master node? Is it possible to change that? |
Hi @hakehuang , this configuration file for frdm_k64f can be easily found at zephyrproject/zephyr/samples/net/gptp/board , but where is the file related to the mimxrt1020_evk configuration? Where do we must add these config parameters for mimxrt1020_evk? Thanks in advance! |
@santi681 , you can look the the eth_mcux.c code and find the CONFIG_PTP_CLOCK_MCUX, this is the option to enalbe ptp time stamp. |
the SDK exmaple only to use to check whether there is a board issue. it only examples to add timestamp to udp package, which is not a ptp example. In zephyr gptp case, according to the gptp protocal, when you connect two ptp instance, they will send out pdelay_req and then choose to be master or client, then mast starts to send sync packet. I do not have a workable linux host as ptp master. I tried to connect two boards, but it looks buggy as I reported in #29599 |
Clear with that, thanks @hakehuang ! |
Thanks for your replies @hakehuang ! I simply defined these parameters at the beginning of the eth_mcux.c code if this is what I was supposed to do. Defining these config parameters doesn't make any difference to the results. In fact PTP seems to be working properly because slave_time.second are properly synchronized, the problem still remains on getting the slave_time.nanosecond, which is still zero at any time. |
You should not do that, but set the options in your applications prj.conf file. Some other source file might use these options too in which case they would not be set in all the needed places. |
the main missing here is in the HAL driver see the this enables the HAL driver to output ptp time stamp. |
CMakeLists.txt file: I can see these config parameters are already defined. Does it make sense to be able to get the seconds and not the nanosecond? Are the requirements for both parameters the same? |
it depends on CONFIG_PTP_CLOCK_MCUX, so you need define it in boards just like frdm_k64f does, and this in HAL driver enables the hardware ptp timestamp |
|
Hello everyone. I wanted to give a quick update and confirm that we see the same problem on the RT1050EVK. I believe we are configuring prj.conf correctly and have also confirmed that the HAL driver is being referenced. We are preparing a complete package of code, working on the frdm_k64f, NOT working on 1020 and 1050, to be able to share here. |
Thanks kevin. please share your code. below are my debug log on rt1060 which I think should be same |
|
I am seeing the same issue when running the gPTP sample on the |
|
We have some new information. Unfortunately, due to continuing logistical difficulties we still don't have the K64-FRDM at our location, and we are not able to run the code with the latest modifications on it. We ARE able to run the same code on the mimxrt1050_evk and mimxrt1020_evk, however, and we see the same behavior as mentioned in all the previous posts. Beyond this, we are starting to looking at the guts of zephyr/modules/hal/nxp/mcux/drivers/imx/fsl_enet.c, and we think we see some things that aren't working as they should be. First, a couple of links to our code in github: The general issue continues. When the EVK running Zephyr as a slave syncs with our PC running Avnu gptp, we see the seconds field pop into the correct values, but the nanoseconds field is stuck at 0. Recently, we have started to extract data from what we believe to be the source of the nanosecond field, the ENET_ATVR register (see page 2155 of the i.MX RT1050 Processor Reference Manual, Rev. 4 12/2019). We do this in fsl_enet.c (see ainguraXmarquiegui/hal_nxp@02947be ), and print it in our application. The exact changes can be seen here: With this code compiled and running, this is what we see on the console of both the 1020EVK and 1050EVK: As you can see, the ATVR register address is correct. The ATVR register value, however, is always reading 0x0. We believe this may be caused by the ATCR Reset Timer bit, which we should see getting automatically cleared. We are always reading 0xA91, but think that we should be seeing an 0x8 instead of 0xA. |
I check your https://github.com/ainguraXmarquiegui/zephyr/tree/feature/aingura_gptp_tests but I did not see you modify the west.yml to direct your hal code. how do you do that? do you run west update before your build? |
|
Yes. We ran west update initially, and we are not changing between platforms, it is always either RT1050 or RT1020. My colleague @ainguraXmarquiegui ran west update just before creating the fork. |
|
can you have a picture on your real boards connection? and if possible can I have your linux host boards shipped to me? |
|
Hello @hakehuang and everyone listening in, To anyone trying our code, please do a fresh checkout of the latest commit at: https://github.com/ainguraXmarquiegui/zephyr/tree/feature/aingura_gptp_tests Thanks to our friends at EBV, we got our hands on a FRDM-K64F and were able to run the same code on all three boards, and have confirmed that the FRDM works as expected, and the i.MXRT boards do not--and we are quite sure that it is related to the Reset Timer bit in the ATCR register. First, as requested, this is the test setup, including are how the cables are hooked up:FRDM-K64F, connected via USB to my PC (via the SDA USB port for the console UART), and with an Ethernet connection directly from the board to the Ethernet interface of my PC:
MIMXRT1020-EVK, connected via USB to my PC (via the J13 USB port for the console UART), and with an Ethernet connection directly from the board to the Ethernet interface of my PC:
MIMXRT1050-EVKB, connected via USB to my PC (via the J28 USB port for the console UART), and with an Ethernet connection directly from the board to the Ethernet interface of my PC:
My PC is nothing special. It does have an Intel I219-LM LOM Ethernet interface with hardware timestamping, which is very useful for gPTP, but most corporate PCs with NICs from Intel or Broadcom should have this capability: As suggested in the README in the gptp sample application, I have installed OpenAvnu/gPTP: https://docs.zephyrproject.org/latest/samples/net/gptp/README.html#setting-up-linux-host As specified I have increased the neighborPropDelayThresh in the gptp_cfg.ini file, in this case to 1000000, to absorb any imprecise adjustment in the PHY delay on the Zephyr side. Here is the gptp_cfg.ini file we are using: gptp_cfg.ini.txt Here is what we are seeing, with the SAME code, running in the SAME way, on all three boardsFRDM-K64FNote how the Reset Timer bit in the ATCR register is 0, the nanoseconds field is getting filled, and the application is able to display nanoseconds:
MIMXRT1020-EVKNote how the Reset Timer bit in the ATCR register is 1, the nanoseconds field is NOT getting filled, and the application is reading 0 nanoseconds:
MIMXRT1050-EVKBNote how the Reset Timer bit in the ATCR register is 1, the nanoseconds field is NOT getting filled, and the application is reading 0 nanoseconds:
It is not going to be practical for me to send you my principal PC. If it would be VERY helpful and DRASTICALLY reduce the time it will take to resolve this issue, I can find another host device with hardware timestamping support (for OpenAvnu gptp), and send it to you. Let's get this thing solved! Thanks everybody! |
|
I just wanted to say that we will let the bot close this issue because neither we nor anyone else seem to have bandwidth or the will to work on it anymore, but this problem has definitely NOT been solved. To have a stable and robust gPTP implementation on i.MXRT, someone needs to implement a servo to keep the PHC locked and keep the corrections smooth. We have "solved" this issue by giving up on Zephyr and going back to FreeRTOS. Much to my chagrin, because I really like the features and community Zephyr brings to high-performance microcontrollers. |
|
@kevin137, I've removed the stale and reopened the issue. |
|
We've given up. We will let the bot close the issue. |
|
@dleach02 I will follow up this fix and retest the gptp performance as well as stabilities. this fix looks good. the gptp of nxp driver now can achieve the same result as ST boards. and it can endure 24 hours testing without lost sync. but below code still has risk https://github.com/zephyrproject-rtos/zephyr/blob/main/drivers/ethernet/eth_mcux.c#L863 if ts is wrapped at the boundary of 1 second, such as 99999999 -> 000000001, then we need compensate this gap. below code are recommend by Seb Laveze
after 24 hours testing to summary here:
@kevin137 , @DerekSnell , @dleach02 , @lomn please take chance to check this issue in latest zephyr code, with |
|
Closing the issue. |
Description of bug
Problem with the gptp_event_capture() function in the gPTP demo in ZephyrOS on the RT1020EVK: time.second member of net_ptp_time struct is updated correctly, but time.nanosecond is NOT. The same function in the same demo on the FRDM-K64F works correctly.
Background
We are trying to use gPTP (IEEE 802.1AS) on the i.MXRT platform to synchronize timestamps across devices in a sensor network. All the hardware documentation for the i.MXRT and App Notes such as AN12149 (https://www.nxp.com/docs/en/nxp/application-notes/AN12149.pdf) indicate that this should be possible. We are interested in using ZephyrOS because we feel it should work well with the way we develop embedded systems, which are at the moment all based on Linux and Yocto. Zephyr seems to have good network support, and gPTP support, at least on some platforms (https://docs.zephyrproject.org/latest/reference/networking/gptp.html). To kick-start this process, and to evaluate whether gPTP/ZephyrOS/i.MXRT is a good technical solution for synchronization, we asked a graduate student at Universitat Politècnica de Catalunya, Santi Prats, to help us.
Santi advanced significantly, first on installing the SDK and tools, then with compiling Zephyr and applications for i.MXRT and Kinetis, also with PTP and gPTP synchronization purely in Linux. Then he encountered a problem which stumped him and us.
To Reproduce
On the FRDM-K64F board (https://docs.zephyrproject.org/latest/boards/arm/frdm_k64f/doc/index.html), he is able to extract timestamps from the gPTP system with nanosecond resolution. But on the RT1020EVK (https://docs.zephyrproject.org/latest/boards/arm/mimxrt1020_evk/doc/index.html), using the same code, with the same libraries and tools, the nanosecond member of the net_ptp_time struct is NOT being filled, and therefore we are only able able to read the PTP Hardware Clock with a resolution of seconds. Reproducing the issue is as simple as adding a few extra LOG_INF() calls to the gptp demo (see below), compiling it for the RT1020-EVK, flashing it, and running it.
Lightly modified demo code
Build
Run
Expected behavior
The expected behavior is what happens running the same code on the FRDM-k64f:
Note that on the FRDM-k64f, the nanosecond member IS being updated.
Impact
We are unable to proceed with the development of the time synchronization element of our new sensor solution because of this issue. Santi is attempting to make equivalent functionality work in FreeRTOS to continue with his academic pursuits, but this is not a great solution for us, because other team members are developing other components of the sensor firmware in Zephyr. We the just getting started with Zephyr; we like what we see so far, but timestamp synchronization with gPTP is at the very heart of what we are trying to do with our sensors--if we can not make it work, we will need to change our strategy. We reached out to NXP through our distributor EBV, and they have confirmed that the gPTP stack and demo should work in Zephyr on the RT1020-EVK. They encouraged us to report the issue here.
GDB / console output
Using the gdb debugger, Santi isolated the differences in the progress of the call to gptp_event_capture to a single line of code that differs between the FRDM-K64F and the MIMXRT1020-EVK. The relevant part of his (attached) document describing the problem follows:
GDB debugging
To investigate the differences between what is happening on the FRDM-K64F and the MIMXRT1020-EVK, the GDB debugger was used. The following was executed from the Zephyr directory:
We enter into "debug mode"
A position is set in the moment just before the execution of
At this instant,
Both return 0. We now step through to see execution of gptp_event_capture()
Before the execution finishes, and analyzing the output of (gdb) step, it is seen that execution passes through the following files and in this order:
Finally,
Now, in the display of the time.seconds member, a correct value is seen, while the time.nanoseconds member still reads 0:
In conclusion, we are seeing the desired behavior on the FRDM-K64F; we are able to read timestamps down to the nanosecond. The same application code, making the same api calls on the RT1020 is not returning nanoseconds, however, and it all seems to come down to line 62 in gptp_user_api.c
Environment:
Zephyr6.pdf
The text was updated successfully, but these errors were encountered: