Add multi-object tracker

[AlexeyAB]

Add multi-object tracker (dynamic array of single object trackers):

• SORT-tracker: cv::KalmanFilter + Hungarian algorithm https://en.wikipedia.org/wiki/Hungarian_algorithm and https://arxiv.org/abs/1602.00763 and https://github.com/mcximing/sort-cpp

• cv::TrackerCSRT() (CSRDCF++) - OpenCV-contirb

• Object detection task and appearance embedding task simutaneously in a shared neural network - source: https://github.com/Zhongdao/Towards-Realtime-MOT pdf: https://arxiv.org/abs/1909.12605v1

• SiamRPN - first places in Real-time: https://arxiv.org/abs/1808.06048 and https://github.com/foolwood/DaSiamRPN
  

---

Accuracy:

1. Robustness (number of times a is reinitialized)
2. Accuracy (average overlap while tracking)
3. Combination of basic measures (EAO) - the VOT primary measure is the Expected Average Overlap (EAO) – a principled combination of accuracy and robustness.

Page 16: https://prints.vicos.si/publications/files/365

[buzdarbalooch]

thanks for sharing @AlexeyAB . How can we use this, with yolo detection. secodly, how different is it to already exisiting (OPTFLOW AND KALMAN FILTERING)

[tdurand]

It's really great you are working on baking this into your YOLO fork, I had good results using a very simple one: https://github.com/bochinski/iou-tracker , paper: http://elvera.nue.tu-berlin.de/files/1547Bochinski2018.pdf

I implemented a flaky node.js version: https://github.com/tdurand/node-moving-things-tracker

[frunika]

It's really great you are working on baking this into your YOLO fork, I had good results using a very simple one: https://github.com/bochinski/iou-tracker , paper: http://elvera.nue.tu-berlin.de/files/1547Bochinski2018.pdf

I implemented a flaky node.js version: https://github.com/tdurand/node-moving-things-tracker

Had good experience too. Overall, it's ultra-fast!

[Nuzhny007]

Hi!
I and Smorodov working with multi-object tracker: https://github.com/Smorodov/Multitarget-tracker
We use your fork as one of detector. And if you do not mind, then I could transfer part of the code to this project.

[AlexeyAB]

@Nuzhny007 Hi,

What is the most accurate multi-object tracker you could find and used?

[Nuzhny007]

I'm working only with C++ code with real time systems. And Hungarian algorithm or version with bipartite graph is not bad. And also i'm used more simple algorithms but they are not robust.

A good online algorithm is "FollowMe: Efficient Min-cost Flow Tracking with Bounded Memory and Computation": http://www.cvlibs.net/projects/online_tracking/
But it only on Python. It's implementation on C++ in my plans.

The good are "Lifted Multicut" (https://github.com/jutanke/cabbage) and LBM (Smorodov/Multitarget-tracker#125) but they are too slow.

And also in plans SORT with YOLO: https://github.com/humoncy/YOLOv3-SORT-ReID

[AlexeyAB]

@Nuzhny007

Yes, it seems that SORT is very fast and enough accurate algorithm.

I'm working only with C++ code with real time systems. And Hungarian algorithm or version with bipartite graph is not bad.

• What is the different between your Hungarian algorithm and common SORT-algorithm (cv::KalmanFilter + Hungarian algorithm) ?

• Did you try to combine: cv::cuda::SparsePyrLKOpticalFlow + cv::KalmanFilter + Hungarian algorithm ?

• Did you try cv::TrackerCSRT() (CSRDCF++) from OpenCV-contirb?

[Nuzhny007]

1. Our Hungarian + Kalman == SORT. This method was used by many people long before the article on the SORT (2016) appeared.
2. No. I tryed LK optical flow earler but remove it. The visual object trackers (like KCF) works better.
3. Yes. It can use any visual object tracker for objects that on current frame was absent.

For example ( https://github.com/Smorodov/Multitarget-tracker/blob/master/example/examples.h#L177 ):

  TrackerSettings settings;
  settings.SetDistance(tracking::DistJaccard);
  settings.m_kalmanType = tracking::KalmanUnscented;
  settings.m_filterGoal = tracking::FilterRect;
  settings.m_lostTrackType = tracking::TrackCSRT;
  settings.m_matchType = tracking::MatchHungrian;

Here we use SORT: DistJaccard + Kalman + Hungarian. And we use CSRT tracker (TrackCSRT) for frames when object was undetected.

On image trajectory parts with circles means that here was used visual object tracker (KCF, CSRT, STAPLE or...)

[AlexeyAB]

@Nuzhny007

So circles means, that the SSD-detector can't detect object there, and was used object tracker (KCF, CSRT, STAPLE or...).

Is it this video? https://www.youtube.com/watch?v=Qssz6tVGoOc
It seems that

• timing 0:05 - it loses an object (person) in the middle of the road, and re-identifies it
• timing 0:11 - it loses an object (child) for 1-2 seconds

Why SORT (Kalman + Hungarian) and CSRT can't solve this taks, even if these objects are clearly visible and almost do not overlap?

[Nuzhny007]

Here was used not CSRT and classical KCF from OpenCV - it more faster but not so robust as CSRT. And KCF not robust for dynamic object like people, it tracks static background.

[AlexeyAB]

@Nuzhny007

Hi!
I and Smorodov working with multi-object tracker: https://github.com/Smorodov/Multitarget-tracker
We use your fork as one of detector. And if you do not mind, then I could transfer part of the code to this project.

Yes, you can transfer tracking part to this project by using Pull Requests

• to darknet https://github.com/AlexeyAB/darknet/blob/master/src/image_opencv.cpp

• or/and to ./uselib project that uses libdarknet.so - files https://github.com/AlexeyAB/darknet/blob/master/src/yolo_console_dll.cpp and https://github.com/AlexeyAB/darknet/blob/master/include/yolo_v2_class.hpp

I think there can be added some #ifdef to check if we OpenCV is compiled with OpenCV-contrib (KCF, CSRT, STAPLE or...).

or can be added #define in https://github.com/AlexeyAB/darknet/blob/master/Makefile#L9 for example, TRACK_OBJECTS=1 to compile with Tracking functions. We should set this flag TRACK_OBJECTS=1 if we compiled OpenCV with OpenCV-contrib.

[Nuzhny007]

Ok, I'l do it

[sctrueew]

@Nuzhny007 Hi,

Thanks for your work. I have merged this repo with your repo. I'm going to use for traffic sign detection and my camera is front of the car. Sometimes the signs are not recognized in some frames. I want to know, what is the best TrackerSettings for my case?
Is TrackCSRT better than TrackKCF for my case?

Thanks

[LukeAI]

@AlexeyAB What is the status of multi-object tracking in this repo? Was @Nuzhny007 tracker ever integrated?

[AlexeyAB]

@LukeAI Not started yet.

[Nuzhny007]

We have one problem: Multitarget-tracker has GPL license and I can't use it code in darknet. Smorodov (the owner of the Multitarget-tracker) starded a new project TrackerGenerator for generate tracker's code with new license.

[Nuzhny007]

@zpmmehrdad CSRT tracker is better.

[GeekAlexis]

If anyone is interested, I implemented an optimized (real-time) tracker based on Deep SORT that uses Darknet's YOLO models converted to TensorRT: https://github.com/GeekAlexis/FastMOT

[AlexeyAB]

@GeekAlexis Great, thanks!

• As I see it supports yolov4-p6.cfg/weights and DeepSort. Does it also supports yolov4-csp-x-swish.cfg/weights (it just uses Swish-activation)? https://github.com/AlexeyAB/darknet#pre-trained-models

• yolov4-csp-x-swish.cfg - 640x640 - 69.9% [email protected] (51.5% [email protected]:0.95) - 23(R) FPS / 50(V) FPS - 221 BFlops (110 FMA) - 381 MB: yolov4-csp-x-swish.weights
  • pre-trained weights for training: https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-csp-x-swish.conv.192

• Did you try to use SiamRPN++ from OpenCV, and/or compare DeepSort vs SiamRPN++? https://github.com/opencv/opencv/blame/master/samples/dnn/siamrpnpp.py#L6-L13
  Quite challenging case at 5:47 https://youtu.be/PdZrF145lAI?t=348

• Also just wondering how does this compare to FairMONT? https://github.com/ifzhang/FairMOT

[GeekAlexis]

@AlexeyAB

Thanks for your interest. I will support swish CSP weights.

I have not tried to use SiamRPN++. Because my implementation uses KLT (optical flow) with DeepSORT, a possibility is to replace KLT with SiamRPN++. It should improve accuracy but a single object tracker probably doesn't scale well with the number of objects in a real-time MOT system.

FairMOT is a faster extension of Deep SORT because it combines detection and ReID embedding in one network. I have plans to support it if I can convert its YOLOv5 backbone version to TensorRT. The author's implementation is not as optimized and its tracking framework is not as robust to camera motion and occlusion compared to mine.

Also, combining FairMOT with a scaled-YOLOv4 backbone would be nice.