Evaluation of many trackers + FPS check + additional fixes

[Fleyderer]

I've added tracking/eval.py script for evaluation many detectors, reids, trackers at the same time with different FPS.

As a result of command:

python3 tracking/eval.py --yolo-models yolov8n.pt yolo11n.pt --reid-models osnet_x0_25_msmt17.pt --tracking-methods bytetrack deepocsort botsort --source ./tracking/val_utils/data/MOT17/train --verbose

You'll get results and plots of val.py for each combination under runs/eval. For example, plots are:

I've tried to add easy to understand progress bars and also fixed some problems while loading data etc.

[Fleyderer]

BTW, I also just returned from vacation and prepared a new PR 🙂

[mikel-brostrom]

Really nice!

{
    "yolov8n": {
        "osnet_x0_25_msmt17": {
            "bytetrack": {
                "original": {
                    "HOTA": 32.353,
                    "MOTA": 26.286,
                    "IDF1": 35.989
                },
                "5": {
                    "HOTA": 29.004,
                    "MOTA": 23.054,
                    "IDF1": 31.559
                },
                "10": {
                    "HOTA": 31.972,
                    "MOTA": 25.866,
                    "IDF1": 36.412
                },
                "15": {
                    "HOTA": 32.631,
                    "MOTA": 26.706,
                    "IDF1": 37.186
                },
                "20": {
                    "HOTA": 32.943,
                    "MOTA": 26.358,
                    "IDF1": 37.473
                }
            },
            "deepocsort": {
                "original": {
                    "HOTA": 29.931,
                    "MOTA": 23.195,
                    "IDF1": 33.956
                },
                "5": {
                    "HOTA": 25.282,
                    "MOTA": 17.378,
                    "IDF1": 25.458
                },
                "10": {
                    "HOTA": 28.084,
                    "MOTA": 21.197,
                    "IDF1": 30.739
                },
                "15": {
                    "HOTA": 31.02,
                    "MOTA": 23.993,
                    "IDF1": 35.447
                },
                "20": {
                    "HOTA": 28.885,
                    "MOTA": 22.551,
                    "IDF1": 32.295
                }
            },
            "botsort": {
                "original": {
                    "HOTA": 24.819,
                    "MOTA": 19.46,
                    "IDF1": 23.58
                },
                "5": {
                    "HOTA": 24.226,
                    "MOTA": 17.201,
                    "IDF1": 23.504
                },
                "10": {
                    "HOTA": 25.586,
                    "MOTA": 19.196,
                    "IDF1": 24.925
                },
                "15": {
                    "HOTA": 25.154,
                    "MOTA": 19.242,
                    "IDF1": 24.23
                },
                "20": {
                    "HOTA": 25.524,
                    "MOTA": 19.376,
                    "IDF1": 24.598
                }
            }
        }
    },
    "yolo11n": {
        "osnet_x0_25_msmt17": {
            "bytetrack": {
                "original": {
                    "HOTA": 30.292,
                    "MOTA": 22.719,
                    "IDF1": 32.475
                },
                "5": {
                    "HOTA": 26.048,
                    "MOTA": 19.9,
                    "IDF1": 27.156
                },
                "10": {
                    "HOTA": 30.705,
                    "MOTA": 22.225,
                    "IDF1": 33.701
                },
                "15": {
                    "HOTA": 31.448,
                    "MOTA": 22.997,
                    "IDF1": 34.726
                },
                "20": {
                    "HOTA": 30.927,
                    "MOTA": 22.844,
                    "IDF1": 33.815
                }
            },
            "deepocsort": {
                "original": {
                    "HOTA": 27.579,
                    "MOTA": 19.677,
                    "IDF1": 29.721
                },
                "5": {
                    "HOTA": 24.059,
                    "MOTA": 14.89,
                    "IDF1": 23.496
                },
                "10": {
                    "HOTA": 26.63,
                    "MOTA": 17.808,
                    "IDF1": 27.358
                },
                "15": {
                    "HOTA": 29.318,
                    "MOTA": 20.924,
                    "IDF1": 32.188
                },
                "20": {
                    "HOTA": 27.338,
                    "MOTA": 19.386,
                    "IDF1": 29.32
                }
            },
            "botsort": {
                "original": {
                    "HOTA": 22.59,
                    "MOTA": 16.261,
                    "IDF1": 19.823
                },
                "5": {
                    "HOTA": 21.834,
                    "MOTA": 14.395,
                    "IDF1": 20.21
                },
                "10": {
                    "HOTA": 22.991,
                    "MOTA": 15.964,
                    "IDF1": 20.827
                },
                "15": {
                    "HOTA": 23.031,
                    "MOTA": 16.203,
                    "IDF1": 20.563
                },
                "20": {
                    "HOTA": 23.415,
                    "MOTA": 16.315,
                    "IDF1": 20.939
                }
            }
        }
    }
}

Love the FPS feature. Maybe it should be an argument? Would help with experimentation

[Fleyderer]

Maybe it should be an argument?

I did this:

fps_parser = parser.add_mutually_exclusive_group(required=False)
fps_parser.add_argument('--fps-check', dest='fps_check', action='store_true',
                        help='Check dataset working on different FPS values.')
fps_parser.add_argument('--no-fps-check', dest='fps_check', action='store_false',
                        help='Do not check dataset working on different FPS values.')
parser.set_defaults(fps_check=True)

parser.add_argument('--fps-values', nargs='+',
                    default=FPS_DEFAULT_SET, type=str)

So, if you don't need to check FPS, you pass --no-fps-check, or if you want to use own values, you can do --fps-values 5 6 7 8 and we'll check 5 6 7 8 FPS instead of 5 10 15 20, which are default

UPD: Perhaps this should be described in more detail in the README.md, I didn't spend much time because I wasn't sure if you would merge it or not 🙂