Doubts about the effectiveness not being as good?

[WesternTrail]

Recently, I used the pre trained weights of resnet18 provided in the work of this paper to migrate to downstream change detection networks. And my network is more complex than the change detection network mentioned in this paper, resulting in: the pre trained weights using the weights have bad performance than directly using Imagenet's pre trained weights:

1. Freeze the backbone feature extraction network and only train the subsequent parts: the F1 score of task:change detection in WHU-CD has a performance degradation of 1-2 points.

2. Do not freeze the backbone, fine tune the entire network, F1 indicators are similar, but there is still a performance degradation of 0-0.5 points.

How explain this phenomenon? Is it because the network of the downstream tasks in the paper is too simple to learn more patterns, so the use of remote sensing data has a significant increase in pre training weights?