[Improvement?] Have you tried a better polygon fit?

[molesquirrel]

First off, amazing work you've done. Using your model for a small project to record existing routes for a local gym. The model works amazingly well.

In your work to train the model, did you consider or try using what I believe are called segmentation masks to get a better fit for the holds? In my particular project, I am hoping to, after registering a route, desaturate the other holds/wall, so that the route stands out more.

If you gave it a shot but it did not come up with results that you were happy with, I'd love to know about it, otherwise I may try training a new model using your training data (although I am wholly unfamiliar with ml training, hence my reason for reaching out to you first.)

[mkurc1]

Thanks a lot! I really appreciate that this project is useful for you.

Yes, I considered it, but didn't try. I thought of it as you. It should bring better results, but it demands more time to prepare data. For the research goal box mask, it was enough.

[molesquirrel]

That's completely understandable. I've been very happy with the level of precision currently available, just had the tighter fit polygons as nice to have for a possible future upgrade.

[molesquirrel]

Hello again. I've been working on a yoloV8 segmentation model that bootstraps off of your training data for your model. It's still missing about 15% of holds, and has some false positives, but I'm continuing to refine it. As for fitting on the holds, it does an exquisite job.

I found that the two models work extremely well in tandem: your model to identify the bounding boxes, then feeding that into my v8 model to get the polygonal bounds.

I wanted to mention this to you in case you are interested in the model/training data once I finish. If so, I can put it up somewhere on github for you.

[mkurc1]

Hi! Thanks for your suggestions, and I would love to see your improvements.

[molesquirrel]

I hope you do not mind me treating this as more of a conversation. I'll share my progress with you up until now.

I initially used SAM to take your dataset, build polygons inside of the bounding boxes you already have, and then use that to train a yoloV8 (yoloV9 does not have segmentation). The dataset creation went extremely well, but I cannot yet get a model that is satisfactory. Running the v8 model alongside yours, it will pick up 90% of the holds that your model gets, but adds a few false positives.

It works really well if your situation will allow you to have masked (segmentation) data on most of your holds. My setup currently colors holds on the wall, and it does a:
if ( maskExists) colorWithMask() else colorwithBBox()

kinda thing. Keeps the false positives out, and adds a lot of useful data to your model's output. Not sure how much more I can eek out of that model without having to run it for several days on my macmini (M2)

I am currently working on a different approach. I'm using the mobile SAM model now. It takes your v9 results, then uses the SAM to build the polygon data inside of the box (similar to how I rebuilt your training dataset). It matched 100% of your holds, but it takes significantly longer (I'd say about 60-80 seconds for a 4000x3000px image with 150 holds on it).

My ultimate goal is to get the v8 model working well enough so it can survive on its own, but I'm not sure I have the patience to change some settings then wait 24-48 hours to see if there are any improvements.

I can, of course, share the v8 model file, as well as the scripts I'm using to a)run the v8/v9 next to each other as well as the b)run through v9, then through sam). Let me know what you'd like to see.