[Bug]: unable to import lemmy settings from one instance to another

[nene2999]

Requirements

• Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• Did you check to see if this issue already exists?
• Is this only a single bug? Do not put multiple bugs in one issue.
• Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

i tried backing up my lemmy subs and settings from one instance (lemm.ee) to another (thelemmy.club). I exported the file (a json-file) from settings. But I was unable to import the file to another instance. I keep getting an error message telling me that the file is too big. It was 'just' 70 kb.

Steps to Reproduce

1. go to settings, export settings, download json-file
2. still in settings import json-file
3. get an error-message, error importing settings, error too many items
4. i try it again i get: rate limit error and please try again

Technical Details

i get the same error on both android 8 and windows 10 pc

Version

BE: 0.19.1-rc.1

Lemmy Instance URL

thelemmy.club and lemm.ee

[dessalines]

TooManyElements look like it has a 1k element limit. @Nutomic Maybe that should be removed, and we should just rely on nginx body upload limit.

[Nutomic]

The nginx upload limit is a few megabytes due to image uploads. In terms of json that would be millions of items which would easily overload a server. So I would definitely keep the limit, but increase it. Its also unfortunate that the rate limit applies even if the request threw an error, so you have to wait a day to try again.

@nene2999 I just checked the backup for my account, and its only 6 kb with roughly 180 items. Can you say which part of your backup is so big, and post the result of cat lemmy_user_settings_20231220T112457.674Z.json | jq | wc -l to get a size estimate? As a workaround for now you can delete parts of the backup to make it smaller before importing.

[nene2999]

i guess where i know where my problem is. I'm a bit of a digital hoarder. I have hundreds of subscriptions. Is there a way to export them all or do i have to clean up the json-file?

[Nutomic]

Hundreds should be no problem, the limit is 1000 items. What you can do is delete parts of your backup so that it is under the limit. Then make several imports with different parts of the backup file (you have to wait a day each time due to rate limits). Maybe we should make the limit configurable for admins.

[MaximilianKohler]

I also get an error, but mine just says "error importing settings".

I had edited the .json in a few different ways. First I removed the entries that I didn't want imported (everything except blocked communities), and then I tried changing some of the values to null for the entries I didn't want to be overwritten or added. Apparently, these edits are what caused the error, since it worked with a completely unedited version.

Maybe I should put this suggestion in another issue, but the Import/Export Settings section should have information about how it handles adding or overwriting data. I read on a 3rd party site that Lemmy's import is additive in nature and doesn't overwrite anything, and that's what gave me the confidence to try this. But that information should be in the Lemmy settings area, and I also don't want certain things added to blank entries. For example, nearly everything in the first section of the json before it starts the followed_communities.

"display_name":xxx,"bio":xxx,"avatar":xxx,"banner":xxx,"matrix_id":xxx,"bot_account":false,"settings":{"id":xxx,"person_id":xxx,"email":"xxx",

Etc.

[dessalines]

If its the correct format, it shouldn't have an issue. Instead of null, you'd just leave off that field, since rust treats it as an Option

[MaximilianKohler]

I just tested this again. I removed:

"id":1234,"person_id":1234,"email":"[email protected]",

"local_user_id":123,

And I get error importing settings.

I then changed the values to null "id":null,"person_id":null,"email":"null" and still got the error.

I then changed the values to "id":111,"person_id":111,"email":"[email protected]", and it worked.

This is doable now that I know exactly how to do it, but it should be easier.

For example, if I have one of these set on one account "display_name":null,"bio":null,"avatar":null,"banner":null,"matrix_id":null, I don't want them being imported to the other account. I really only want to transfer the followed & blocked lists, but if I remove everything before "followed_communities" I get the error.

It would also be nice if there was an overwrite option. For example, if I wanted to completely mirror another account or quickly remove all my blocks, I don't see an easy method to do that.

EDIT: For some reason, it didn't import the "blocked instances". I know there can be a delay, but there weren't many changes and it's been 3 hours.

[Nutomic]

@MaximilianKohler I made a PR to fix the partial importing. Blocked instance import should work, can you check the server logs for relevant errors?

And regarding overwrite option it should first remove all community follows, saved posts etc before importing? Not sure how that should work if a field is missing from the backup. Anyway please open a new issue for that feature request.

[MaximilianKohler]

I'm importing it to lemmy.world, and I'm not an admin, so I don't think there's a way for me to check the server logs?

[Nutomic]

I tested on ds9.lemmy.ml and blocked instance import worked fine. Maybe there was a fix in 0.19.4 or .5

[MaximilianKohler]

Perhaps it has to do with the edits I made, because I've done it successfully before, but after editing it this time, it said it successfully imported but doesn't appear to have done so. I tried twice, a day apart. I only changed the IDs and email to 111.

"display_name":null,"bio":null,"avatar":null,"banner":null,"matrix_id":null,"bot_account":false,"settings":{"id":111,"person_id":111,"email":"[email protected]","show_nsfw":true,"theme":"darkly-compact","default_sort_type":"Hot","default_listing_type":"All","interface_language":"browser","show_avatars":true,"send_notifications_to_email":false,"show_scores":true,"show_bot_accounts":true,"show_read_posts":true,"email_verified":true,"accepted_application":true,"open_links_in_new_tab":true,"blur_nsfw":true,"auto_expand":false,"infinite_scroll_enabled":false,"admin":false,"post_listing_mode":"List","totp_2fa_enabled":false,"enable_keyboard_navigation":false,"enable_animated_images":true,"collapse_bot_comments":false},"vote_display_mode_settings":{"local_user_id":111,"score":false,"upvotes":true,"downvotes":true,"upvote_percentage":false},