real time super slow

[tiralonghipol]

Hi. I calibrated my camera with Kalibr, and setup everything.
I was wondering which parameters can I act on in order to speed up the performance for real time..
Right now, seems like rovio keeps doing calculations on the first frame on startup and if I move the camera the frame still fixed.
I'm using an i7 pc, and the cpu in not saturated..
any hint?
Thank you!

[helenol]

Rovio should run real-time at about 20-30 Hz using about ~20% CPU on a standard laptop CPU with default parameters out of the box. Make sure you compile in Release mode (and not debug) and that the data is piped in correctly, and that you're feeding in IMU data that's time synchronized to the camera. Rovio will not work without IMU data.

[tiralonghipol]

Ok then I'm assuming some bad camera/imu sync.. how to check that? any suggestion?
thank you for your reply, btw

[helenol]

Maybe the easiest thing to do is to first check that it's running correctly on the euroc dataset, and in real-time. This will rule out that it's anything other than your data.
Probably the easiest way to check your camera-IMU time sync is to run kalibr ( https://github.com/ethz-asl/kalibr ) for cam-IMU calibration. If you're getting huge reprojection errors, or the motions aren't matching, then this is your issue.

[tiralonghipol]

ok I solved the problem of the 'fixed initial frame' (really thank you!!)
I've repeated the calibration with Kalibr many many times, and I think this is the best result i can obtain with such a cheap imu like the mpu6050.. what do you think?

Calibration results

Normalized Residuals

Reprojection error (cam0): mean 0.588927292038, median 0.488037950524, std: 0.426175123678
Reprojection error (cam1): mean 0.592739630203, median 0.479429252212, std: 0.469319461226
Gyroscope error (imu0): mean 1.17970688177, median 0.910501933594, std: 1.10498299154
Accelerometer error (imu0): mean 0.552302048441, median 0.481720104149, std: 0.334006283476

Residuals

Reprojection error (cam0) [px]: mean 0.588927292038, median 0.488037950524, std: 0.426175123678
Reprojection error (cam1) [px]: mean 0.592739630203, median 0.479429252212, std: 0.469319461226
Gyroscope error (imu0) [rad/s]: mean 0.080011613555, median 0.0617532456391, std: 0.0749435927439
Accelerometer error (imu0) [m/s^2]: mean 0.0374589474289, median 0.0326718470585, std: 0.0226534083097

Transformation (cam0):

T_ci: (imu0 to cam0):
[[-0.99996808 -0.00446951 -0.00662233 0.08393679]
[ 0.00445056 -0.99998597 0.00287312 -0.01480612]
[-0.00663508 0.00284356 0.99997394 -0.06664928]
[ 0. 0. 0. 1. ]]

T_ic: (cam0 to imu0):
[[-0.99996808 0.00445056 -0.00663508 0.08355779]
[-0.00446951 -0.99998597 0.00284356 -0.01424124]
[-0.00662233 0.00287312 0.99997394 0.06724594]
[ 0. 0. 0. 1. ]]

timeshift cam0 to imu0: [s] (t_imu = t_cam + shift)
0.0

Transformation (cam1):

T_ci: (imu0 to cam1):
[[-0.99978718 -0.00505324 -0.02000146 -0.0151341 ]
[ 0.00495019 -0.99997424 0.00519807 -0.01584605]
[-0.02002721 0.00509795 0.99978644 -0.06499639]
[ 0. 0. 0. 1. ]]

T_ic: (cam1 to imu0):
[[-0.99978718 0.00495019 -0.02002721 -0.01635413]
[-0.00505324 -0.99997424 0.00509795 -0.01559077]
[-0.02000146 0.00519807 0.99978644 0.06476218]
[ 0. 0. 0. 1. ]]

timeshift cam1 to imu0: [s] (t_imu = t_cam + shift)
0.0

Baselines:

Baseline (cam0 to cam1):
[[ 0.99991031 0.00054608 -0.01338164 -0.09994716]
[-0.00051506 0.99999717 0.00232161 -0.000842 ]
[ 0.01338287 -0.0023145 0.99990777 0.00048916]
[ 0. 0. 0. 1. ]]
baseline norm: 0.0999518992169 [m]

Gravity vector in target coords: [m/s^2]
[-0.09245357 -9.80600617 0.04602388]

Calibration configuration

cam0

Camera model: pinhole
Focal length: [460.8359042600893, 460.59651287135466]
Principal point: [354.44182102814153, 202.07767726099678]
Distortion model: equidistant
Distortion coefficients: [-0.10387145094103571, -0.03249349371414373, 0.0522382910564682, -0.030513973256192786]
Type: aprilgrid
Tags:
Rows: 6
Cols: 6
Size: 0.088 [m]
Spacing 0.0264 [m]

cam1

Camera model: pinhole
Focal length: [458.97504833502177, 458.4133972042093]
Principal point: [366.09274574886456, 235.02411089106826]
Distortion model: equidistant
Distortion coefficients: [-0.11659497785718727, 0.03624322804994296, -0.08002202623988071, 0.05188458182376746]
Type: aprilgrid
Tags:
Rows: 6
Cols: 6
Size: 0.088 [m]
Spacing 0.0264 [m]

IMU configuration

IMU0:

Model: calibrated
Update rate: 184.0
Accelerometer:
Noise density: 0.005
Noise density (discrete): 0.0678232998313
Random walk: 0.00433
Gyroscope:
Noise density: 0.005
Noise density (discrete): 0.0678232998313
Random walk: 0.00266
T_i_b
[[ 1. 0. 0. 0.]
[ 0. 1. 0. 0.]
[ 0. 0. 1. 0.]
[ 0. 0. 0. 1.]]
time offset with respect to IMU0: 0.0 [s]
<

[ZacharyTaylor]

The output timeshift from kalibr is currently 0.0s. This indicates that it was run without time calibration. Could you rerun it with the --time-calibration flag as this will let us know if there is any offset in timestamping between your camera and IMU

[tiralonghipol]

I feel really stupid.. thank you!
What if the seller declared "hardware camera-imu sync" ?
Does the --time-calibration flag still valid ?

Adding that flag, btw, leads to a new error:
Exception in thread block: [aslam::Exception] /home/pol/kalibr_workspace/src/Kalibr/aslam_nonparametric_estimation/aslam_splines/src/BSplineExpressions.cpp:447: toTransformationMatrixImplementation() assert(_bufferTmin <= _time.toScalar() < _bufferTmax) failed [1.53259e+09 <= 1.53259e+09 < 1.53259e+09]: Spline Coefficient Buffer Exceeded. Set larger buffer margins!

Before you ask, my imu.yaml file that I used for the second step in the calibration (kalibr) is:

#Accelerometers
accelerometer_noise_density: 0.005    #Noise density (continuous-time)
accelerometer_random_walk:   4.33e-03   #Bias random walk

#Gyroscopes
gyroscope_noise_density:     0.005   #Noise density (continuous-time)
gyroscope_random_walk:       2.66e-03   #Bias random walk

rostopic:                    /imu0      #the IMU ROS topic
# update_rate:                 200.0      #Hz (for discretization of the values above)
update_rate:                 184.0      #Hz (for discretization of the values above)

[ZacharyTaylor]

If the system has accurate hardware timestamping than --time-calibration will output something like 1ms or less of offset verifying that the timestamping is accurate.

That error occurs as the system has a larger time offset than kalibr is expecting (default max of 0.01s). Try setting --timeoffset-padding 0.1 and rerunning. However this already suggest the hardware timestamping has some offset in it. Out of curiosity which sensor are you using?

[tiralonghipol]

Here is the link to the sensor:
https://www.robotshop.com/en/optor-visual-inertial-camera.html

This is really strange:

After Optimization (Results)
==================
Normalized Residuals
----------------------------
Reprojection error (cam0):     mean 0.339600946353, median 0.260765657464, std: 0.289844298467
Reprojection error (cam1):     mean 0.333649057074, median 0.232928501278, std: 0.358334196077
Gyroscope error (imu0):        mean 1.6110380717, median 1.12279369489, std: 1.55530907923
Accelerometer error (imu0):    mean 0.702226789553, median 0.632598330057, std: 0.401069750157

Residuals
----------------------------
Reprojection error (cam0) [px]:     mean 0.339600946353, median 0.260765657464, std: 0.289844298467
Reprojection error (cam1) [px]:     mean 0.333649057074, median 0.232928501278, std: 0.358334196077
Gyroscope error (imu0) [rad/s]:     mean 0.109265918176, median 0.076151573417, std: 0.105486194011
Accelerometer error (imu0) [m/s^2]: mean 0.0476273380974, median 0.0429049062122, std: 0.0272018739181

Transformation T_cam0_imu0 (imu0 to cam0, T_ci): 
[[-0.99928476  0.0338582   0.01683991  0.06379936]
 [-0.03217929 -0.99527841  0.09157172 -0.01407266]
 [ 0.01986085  0.09096432  0.99565608  0.02065697]
 [ 0.          0.          0.          1.        ]]

cam0 to imu0 time: [s] (t_imu = t_cam + shift)
-0.0167532691232

Transformation T_cam1_imu0 (imu0 to cam1, T_ci): 
[[-0.99947848  0.03209441  0.00356489 -0.03643763]
 [-0.0316184  -0.99508185  0.0938743  -0.01489952]
 [ 0.0065602   0.09371263  0.99557767  0.02203061]
 [ 0.          0.          0.          1.        ]]

cam1 to imu0 time: [s] (t_imu = t_cam + shift)
-0.00588901836098

IMU0:
----------------------------
  Model: calibrated
  Update rate: 184.0
  Accelerometer:
    Noise density: 0.005 
    Noise density (discrete): 0.0678232998313 
    Random walk: 0.00433
  Gyroscope:
    Noise density: 0.005
    Noise density (discrete): 0.0678232998313 
    Random walk: 0.00266
  T_i_b
    [[ 1.  0.  0.  0.]
     [ 0.  1.  0.  0.]
     [ 0.  0.  1.  0.]
     [ 0.  0.  0.  1.]]
  time offset with respect to IMU0: 0.0 [s]

[ZacharyTaylor]

So I have bad news, if this sensor is doing hardware triggering / timestamping than the driver is throwing this information away and just stamping things on arrival in software.

Here is the line in the driver it happens at https://github.com/optor-vis/vi-stereo-v1/blob/master/optor_VI_Sensor_SDK_V1.0/SDK/src/optorcam.cpp#L954

The two cameras have different timing offsets because they are sent one at a time and so have different arrival times. While an offset of 1 or 2ms probably wouldn't be a deal breaker with 16ms of offset things will probably diverge pretty much immediately

[tiralonghipol]

Thanks again for your support and your patience!
So now what do you suggest? I think the options are 2: modify the driver o use other sensors, am I right?

[ZacharyTaylor]

If no hardware timestamps exist you don't have that many good options. You can take the values kalibr gives you and code these into the driver to get things a bit more consistent. However, these offsets will change with factors such as your CPU load and the arrival times will usually have a spread of a few milliseconds.

Since you know the images and IMU are running at a fixed rate you can implement a simple kalman filter to smooth out this ripple and try hardcode the offset, though I am unsure how accurate this will get you.

Most people using VI sensing have their own custom solution. Of the commercial solutions I know of that we have tested:
The S360 sensor - https://zurichsense.com/ expensive and made in low numbers, but we have had good results with it
The DUO - https://duo3d.com/ a little pricey and from memory the IMU is heavily filtered. We had one working but not reliably.
Realsense ZR300 - out of production, but maybe you can find one. A little sensitive to shocks and we had to modify the provided driver, but it does work.
ZED Mini - the maplab guys just got one of these. So far it seems like it works though we are still testing

[tiralonghipol]

many thanks again!
Another question (a little off topic): have you ever tried rovio in dark or completely dark environments? or what could be the best camera/sensor for low light condition?

[ZacharyTaylor]

One with a torch attached. Though seriously we have managed to run rovio on images that are so dark a person can barely see whats going on. As a rule of thumb the bigger the sensor and the larger the lens the more light will be captured, however you hit diminishing returns pretty quickly making active illumination a good option for dark enviroments.

[tiralonghipol]

Do you have any rosbag to share? or related video/paper about these experiments?

[ZacharyTaylor]

The darkest flights we have done were probably in the back of the building shown here https://www.youtube.com/watch?v=49WiPHkkhBw

Here is a short section of rovio running on part of one of these flights

[tiralonghipol]

This is reeeeally impressive! and also what I was looking for!
Thanks again

[Razbotics]

What is the status of rovio running with optor camera? I have a optor camera and i have finished with calibration... but i get slow odometry results

[ZacharyTaylor]

If you are getting slow odometry this usually means that rovio has diverged and it is falling behind real time while it tries to find new features in a large region.

Unfortunately the cause of this will be that tho optor camera relies on poor time synchronization through arrival time timestamping. This combined with the fairly low quality IMU means it is unlikely to ever work well with rovio.

[Razbotics]

Hey thanks for the info...though its a bad news for me...well is there any workarounds for better timestamping?....i found that VINS mono is working quite well with optor cam and i heard that rovio works more robust than VINS mono using optor cam..

[Razbotics]

Is there a way to hack trigger pins of optor camera? I want them for hardware synchronization

[ZacharyTaylor]

I believe VINS has online time offset estimation, which allows it to compensate for these sort of issues. I am unsure about what is possible with the optor. I have never used one, just looked through its driver.

[Razbotics]

Yep ViNS mono can compensate that issue and maybe thats the reason VINS works quite well. Do you have knowledge of hardware pins of a camera module? I can see some accessible pins HS, VS , LED out and PCLK on the back of optor cam. I was hoping if i can use one of them for hardware timestamping and synchronisation.

[chenwinki]

@tiralonghipol Hello, could you give me a hint how to get the param of imu like this ,thank you very much:
Accelerometer:
Noise density: 0.005
Noise density (discrete): 0.0678232998313
Random walk: 0.00433
Gyroscope:
Noise density: 0.005
Noise density (discrete): 0.0678232998313
Random walk: 0.00266

[tiralonghipol]

@chenwinki can you be more specific on your question please? what do you mean by 'get the param of imu'
If you are talking about the configuration file (-yaml), you do not get anything, you set it as you like (check che datasheet of your IMU, of course)
hope it helps

[chenwinki]

@tiralonghipol Thank you for you advice! I calibrate imu using imu_utils, and calibrate imu and camera using kalibr. but I got bad result, even when I put my VI sensor static, the result shows the VI sensor are moving(about ten meter in 100s), could you give me a hint about the result?

[tiralonghipol]

The Optor / Loitor VI sensor has many problems, being a cheap sensor. The worst one is that the sync between images and imu is not really hardware
After many tries with different algorithms I decided to switch to the MyntEye camera and I immediately got good results (after proper calibration, of course)
For more details see the discussion above #192 (comment)

[mechaliomar]

@tiralonghipol So, you have been using the mynteye camera successfully!
-I am also planning to use it for 6 DOF pose estimation using ROVIO.
-Can you suggest something before I start ?

BEST REGARDS
UESTC, CHINA

[tiralonghipol]

Hi @mechaliomar
I ended up in building my own visual inertial sensor. Unfortunately the commercial sensors I tested during this year were not as good as declared
Currently i'm using rovio onboard an autonomous drone and the results are really good
https://youtu.be/N7SK1ekzhe0