ManyRelatedField causes n+1 issue on to_internal_value

[kandji-gregg]

Passing many=True as a param for a foreign-key relation causes drf to use a ManyRelatedField instead. But the problem is that when to_internal_value is called for a ManyRelatedField it loops through each item in data and passes it to the child_relation's to to_internal_value. While this works, every item in the list is individually fetched from the DB, even though it could likely could be queried in bulk.

It's most obvious when using a PrimaryKeyRelatedField and for every ID in the relation, it fetches the object using queryset.get(pk=data) for every id, even if it's 1000. Instead, we could easily do queryset.get(pk__in=data) to retrieve the data.

I suggest adding a new optional class method for Fields that bulk queries for the field. Code for to_internal_value in ManyRelated field could look like:

    def to_internal_value(self, data):
        if isinstance(data, str) or not hasattr(data, '__iter__'):
            self.fail('not_a_list', input_type=type(data).__name__)
        if not self.allow_empty and len(data) == 0:
            self.fail('empty')
        if hasattr(self.child_relation.bulk_to_internal_value):
            return self.child_relation.bulk_to_internal_value(data)
        else:
            return [
                self.child_relation.to_internal_value(item)
                for item in data
            ]

[Kangaroux]

Unless I'm misunderstanding, this should be solved by using prefetch_related on the queryset. Without seeing the model or what's being serialized though it's difficult to help

[kandji-gregg]

For related fields you have to pass in a queryset, it does not use the queryset of the serializer.

class ItemSerializer(serializers.ModelSerializer):
    containers = serializers.PrimaryKeyRelatedField(many=True, queryset=Container.objects.all())

[zach-waggoner]

I just ran into this issue. .prefetch_related() does not appear to be a solution, as the issue is that ManyRelatedField calls the child relation's .to_internal_value() in a loop, each time doing queryset.get(), which always performs a new query. I implemented a solution like the one suggested by @kandji-gregg to work around this.

[auvipy]

I would happy to see a PR that fix and don't change any existing behaviour in DRF