KITTI error is higher than your reported numbers

[fishcu]

Hi,

Thanks for making stereo DSO available.

I have evaluated KITTI benchmark (sequences 00 and 05) across 5 runs of stereo DSO (to account for randomness). The resulting errors are higher than what you report, especially the translational drift.
You report the following:
2.5% trans. error, 0.0045 deg/m rot. error on KITTI 00
2.7% trans. error, 0.0035 deg/m rot. error on KITTI 05.
I found the following:
3.6% trans. error, 0.0040 deg/m rot. error on KITTI 00
3.0% trans. error, 0.0043 deg/m rot. error on KITTI 05.

Could you please tell me if you have tuned any parameters to reach better KITTI results? For example, does your coupling factor correspond to the coupling factor of the original publication (which relates the static stereo matching energy to the frame-to-frame energy)?

I have attached my results of KITTI 00 and 05. The graphs that do not show the estimated trajectory correspond to the average across 5 runs.
I have also found that stereo DSO crashes when it loses track.

KITTI_00_kitti_plots_run_avg.pdf
KITTI_00_run0_kitti_plots.pdf
KITTI_05_kitti_plots_run_avg.pdf
KITTI_05_run0_kitti_plots.pdf

[NikolausDemmel]

What paper are you referring to?

[fishcu]

I am referring to the errors reported on the main page (README.md), under "4. Experiments". Since the authors report a range of errors, I have taken the average of the two values reported on the main page for comparison.

BTW, both of these results are worse than the ranking in the official KITTI benchmark of the non-public research implementation, so either the research implementation is somehow better, or it was better tuned for KITTI. I can't link there right now as the website has been down for several days now.

[NikolausDemmel]

Not sure about the errors reported here. Just make sure you don't mix up this repository and https://vision.in.tum.de/research/vslam/stereo-dso, which are independent works. Not sure about what is on the KITTI ranking page.

[fishcu]

Yes, I'm aware of that. However, AFAIK, you / the TUM people have not published source code, right? For that reason I evaluated this implementation here.

[fishcu]

Anyway, these questions remain:

1. Is there a probable reason why my evaluation produces worse results than the ones published on this repository? For example, did the authors of this repository tune some parameters additionally without pushing them to the repo?
2. What are the reasons for a discrepancy of the results reported in the arxiv paper and the ones of this repo? Translational drift is about 3x worse, rot. drift is about 2x worse (although sample size is unknown / very small).

[NikolausDemmel]

Yes, I'm aware of that. However, AFAIK, you / the TUM people have not published source code, right? For that reason I evaluated this implementation here.

Yes, the code is not available unfortunately.

What are the reasons for a discrepancy of the results reported in the arxiv paper and the ones of this repo? Translational drift is about 3x worse, rot. drift is about 2x worse (although sample size is unknown / very small).

I guess there can be many reasons, since it is a different system and a different implementation. I'm not sure if there is a paper describing the system in this repository, so it is hard to compare. I guess what you see on the KITTI benchmark website is Wang et al. as well and not this repo.

[JiatianWu]

Sorry for late reply. I use code KITTI provide to measure the output results. Their method is pretty complicated, maybe you should test the output with KITTI evaluation code.

[JiatianWu]

Also, I implemented stereo DSO by following how stereo LSD does because the official stereo DSO paper has not published when I did this. I always want to reimplement my code but did not find time. You are welcomed to modify my version following the stereo DSO paper.

[fishcu]

@JiatianWu Thank you for the explanations! Interesting to see that this implementation uses stereo-LSD's algorithm!

I have re-implemented the KITTI benchmarks with some python scripts. The plots you see are generated from my scripts. Their method is not that complicated, it's just not documented very well. 😄

In any case, on the website I see only Wang et al.'s results. How did you test your algorithm with KITTI's evaluation code? From my understanding, you have to send them your results personally (and they will put it up in the rankings).

[JiatianWu]

You do not need to send code to get results. KITTI evaluation code is available on their website, just download and run it to get results..

[fishcu]

Thank you JiatianWu, that clears some confusion. Did you run your implementation with the settings/parameters that are currently available in the repository? 2018-04-29 6:39 GMT+02:00 JiatianWu <[email protected]>:

…

You do not need to send code to get results. KITTI evaluation code is available on their website, just download and run it to get results.. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYLnmhjMh9PeLh8MU44RGwLb1QNij8rgks5ttUQDgaJpZM4TjOYk> .

[yunxiaoshi]

Hi @fishcu

Can you share how you did the evaluation and results plotting? I'm new to this field and from my understanding I would need kitti_xx_pose_estimated.txt and kitti_xx_pose_gt.txt, i. e. how do I get the estimated pose from stereo dso and dump them into a text file?

A million thanks if you could shed some light on this!

[Pedrous]

Hey @fishcu,

How were your results with this stereo-dso? I am having a quite disturbing scale drift with the method. I guess it should be because of the baseline. It's just that at the same time with the same parameters I am getting very good results ORB-SLAM2 stereo version. That is why I doubt that my camera parameters could be so wrong. I am processing a dataset that I collected by myself.

How did you rectify the images? I used openCV and calculated the baseline from the projection matrix after rotating the images in to the same plane. But as I told, I get good results for ORB-SLAM2 with the same baseline, so I am curious if you noticed the same effect?