nixos/doc/immich: init

[Scrumplex]

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[Titaniumtown]

LGTM over all. I have a few nits, besides that. Thank you for the informative docs! I'm going to personally follow this guide now haha.

[getchoo]

Mostly some punctuation nitpicks

[Atemu]

@getchoo if you have this many tiny adjustments to recommend, it's usually easier for everyone involved to just PR them against the PR branch.

[getchoo]

Scrumplex#1

[szethh]

hi, I followed this guide and got stuck with some permission problems. inside my container based setup, the database user and owner of the tables was postgres. however in the nixos module, it's immich by default.

if you come across this issue, you can run this command on your db dump:
sed -i 's/OWNER TO postgres/OWNER TO immich/g' ./immich-final.sql

after that, it worked like a charm and all my images, users, faces etc popped up with no issues :)

thanks for writing up this guide!!

[Atemu]

I think I ran into that when I did my manual migration before this guide's existence too. The guide should probably cover that.

[mhdask]

Hi, i followed this guide, and i ran into an issue with the database, it says that the migrations relations already exists:

Oct 01 23:36:38 immichcontainer server[557]: Starting microservices worker
Oct 01 23:36:41 immichcontainer server[557]: [Nest] 557  - 10/01/2024, 11:36:41 PM     LOG [Microservices:EventRepository] Initialized websocket server
Oct 01 23:36:41 immichcontainer server[557]: QueryFailedError: relation "migrations" already exists
Oct 01 23:36:41 immichcontainer server[557]:     at PostgresQueryRunner.query (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
Oct 01 23:36:41 immichcontainer server[557]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Oct 01 23:36:41 immichcontainer server[557]:     at async PostgresQueryRunner.executeQueries (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/query-runner/BaseQueryRunner.js:425:13)
Oct 01 23:36:41 immichcontainer server[557]:     at async PostgresQueryRunner.createTable (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:425:9)
Oct 01 23:36:41 immichcontainer server[557]:     at async MigrationExecutor.createMigrationsTableIfNotExist (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/migration/MigrationExecutor.js:351:13)
Oct 01 23:36:41 immichcontainer server[557]:     at async MigrationExecutor.executePendingMigrations (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/migration/MigrationExecutor.js:129:9)
Oct 01 23:36:41 immichcontainer server[557]:     at async DataSource.runMigrations (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/data-source/DataSource.js:265:35)
Oct 01 23:36:41 immichcontainer server[557]:     at async DatabaseRepository.runMigrations (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/dist/repositories/database.repository.js:191:9)
Oct 01 23:36:41 immichcontainer server[557]:     at async /nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/dist/services/database.service.js:109:17
Oct 01 23:36:41 immichcontainer server[557]:     at async /nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/dist/repositories/database.repository.js:199:23 {
Oct 01 23:36:41 immichcontainer server[557]:   query: 'CREATE TABLE "migrations" ("id" SERIAL NOT NULL, "timestamp" bigint NOT NULL, "name" character varying NOT NULL, CONSTRAINT "PK_8c82d7f526340ab734260ea46be" PRIMARY KEY ("id"))',
Oct 01 23:36:41 immichcontainer server[557]:   parameters: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   driverError: error: relation "migrations" already exists
Oct 01 23:36:41 immichcontainer server[557]:       at /nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/pg/lib/client.js:535:17
Oct 01 23:36:41 immichcontainer server[557]:       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Oct 01 23:36:41 immichcontainer server[557]:       at async PostgresQueryRunner.query (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
Oct 01 23:36:41 immichcontainer server[557]:       at async PostgresQueryRunner.executeQueries (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/query-runner/BaseQueryRunner.js:425:13)
Oct 01 23:36:41 immichcontainer server[557]:       at async PostgresQueryRunner.createTable (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:425:9)
Oct 01 23:36:41 immichcontainer server[557]:       at async MigrationExecutor.createMigrationsTableIfNotExist (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/migration/MigrationExecutor.js:351:13)
Oct 01 23:36:41 immichcontainer server[557]:       at async MigrationExecutor.executePendingMigrations (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/migration/MigrationExecutor.js:129:9)
Oct 01 23:36:41 immichcontainer server[557]:       at async DataSource.runMigrations (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/node_modules/typeorm/data-source/DataSource.js:265:35)
Oct 01 23:36:41 immichcontainer server[557]:       at async DatabaseRepository.runMigrations (/nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/dist/repositories/database.repository.js:191:9)
Oct 01 23:36:41 immichcontainer server[557]:       at async /nix/store/1v2r4l57p8qmsy8nk10gx558s6d2aj2p-immich-1.116.2/dist/services/database.service.js:109:17 {
Oct 01 23:36:41 immichcontainer server[557]:     length: 104,
Oct 01 23:36:41 immichcontainer server[557]:     severity: 'ERROR',
Oct 01 23:36:41 immichcontainer server[557]:     code: '42P07',
Oct 01 23:36:41 immichcontainer server[557]:     detail: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     hint: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     position: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     internalPosition: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     internalQuery: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     where: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     schema: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     table: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     column: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     dataType: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     constraint: undefined,
Oct 01 23:36:41 immichcontainer server[557]:     file: 'heap.c',
Oct 01 23:36:41 immichcontainer server[557]:     line: '1149',
Oct 01 23:36:41 immichcontainer server[557]:     routine: 'heap_create_with_catalog'
Oct 01 23:36:41 immichcontainer server[557]:   },
Oct 01 23:36:41 immichcontainer server[557]:   length: 104,
Oct 01 23:36:41 immichcontainer server[557]:   severity: 'ERROR',
Oct 01 23:36:41 immichcontainer server[557]:   code: '42P07',
Oct 01 23:36:41 immichcontainer server[557]:   detail: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   hint: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   position: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   internalPosition: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   internalQuery: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   where: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   schema: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   table: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   column: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   dataType: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   constraint: undefined,
Oct 01 23:36:41 immichcontainer server[557]:   file: 'heap.c',
Oct 01 23:36:41 immichcontainer server[557]:   line: '1149',
Oct 01 23:36:41 immichcontainer server[557]:   routine: 'heap_create_with_catalog'
Oct 01 23:36:41 immichcontainer server[557]: }
Oct 01 23:36:41 immichcontainer server[557]: microservices worker error: QueryFailedError: relation "migrations" already exists
Oct 01 23:36:41 immichcontainer server[557]: microservices worker exited with code 1
Oct 01 23:36:41 immichcontainer systemd[1]: immich-server.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 23:36:41 immichcontainer systemd[1]: immich-server.service: Failed with result 'exit-code'.

[Atemu]

@mhdask please share some info such as which version you're migrating from.

Also please just try again (wipe and everything), perhaps you just accidentally did something differently.

[mhdask]

@mhdask please share some info such as which version you're migrating from.

Also please just try again (wipe and everything), perhaps you just accidentally did something differently.

I am migrating from 1.115.0 to 1.116.2
I have tried making a new dump, and tried deleting everything within /var/lib/postgres, so there should not exist anything before importing the dump

If i do a SELECT * FROM migrations in the “old” database, a lot of lines come up

[Atemu]

Could you update to 1.116.2 beforehand and then migrate to the module?

[hacscred]

I don't see the point of including human readable instructions for migrations that can also be automated (that's what computers are for).

What happens when these instructions are out of date?

Are you going to add a test to make sure it continues to work? If not, why should thousands of people be taking the time to even read this potentially out of date document in a year or two, after you have moved on to greener pastures ?

I have no doubt these instructions might be useful today and it might be useful to include it today, but I don't see long term value in this contribution.

Write something on the wiki until you can automate it. At that point, potentially document it and find people that want to maintain it. Once that is done, lasting value can happen. Such a script should probably be written as a collaboration of contributors of multiple larger distributions.

[Atemu]

That's a bit too high of a standard IMHO but feel free to create that perfect script you describe that magically knows to handle all the weird little differences between people's deployments and is up to date into all eternity.

[Titaniumtown]

@mhdask I'm sorry to report that I also had this issue. I ended up just deleting the database and redoing everything.

I worked for hours to try and fix it, but unfortunately never got anywhere.

[Scrumplex]

This is odd. TypeORM seems to check if the table exists before actually trying to create it. https://github.com/typeorm/typeorm/blob/e7649d2746f907ff36b1efb600402dedd5f5a499/src/migration/MigrationExecutor.ts#L496

Could this be an issue with the search path?

[mhdask]

Could you update to 1.116.2 beforehand and then migrate to the module?

@Atemu I just tried this, I upgraded the docker version first, and then made a fresh dump from that, but still have the same issue.

@mhdask I'm sorry to report that I also had this issue. I ended up just deleting the database and redoing everything.

I worked for hours to try and fix it, but unfortunately never got anywhere.

While I dont know how many photos you have, I would really prefer not having to do this, as I have around 35k photos, 5k vidoes, which have been index, thumbnailed, machine learned, etc.. 😆

This is odd. TypeORM seems to check if the table exists before actually trying to create it. https://github.com/typeorm/typeorm/blob/e7649d2746f907ff36b1efb600402dedd5f5a499/src/migration/MigrationExecutor.ts#L496

Could this be an issue with the search path?

@Scrumplex Is this something i can check?

[hacscred]

There's no need to bring out the pitchforks. Apparently, we are just in disagreement. Ultimately, all you need to do is convince someone with commit rights to merge it.

I explained how I would like it to work, if at all. That's all.

I don't know whether there's any RFC that defines such things, but the longer it exists, the more sensitive maintainers have become for the long term viability of changes, I think.

This is not an invitation to further discuss, since your position is already clear.

[Scrumplex]

This is odd. TypeORM seems to check if the table exists before actually trying to create it. typeorm/typeorm@e7649d2/src/migration/MigrationExecutor.ts#L496
Could this be an issue with the search path?

@Scrumplex Is this something i can check?

It would be interesting to see if there are any unexpected differwnces between the original database dump from the container and your current state. Perhaps you could try to dump your current NixOS database and diff the two files. Obviously the media paths are going to be different, but perhaps something else is wrong too?

[Titaniumtown]

@mhdask

While I dont know how many photos you have, I would really prefer not having to do this, as I have around 35k photos, 5k vidoes, which have been index, thumbnailed, machine learned, etc.. 😆

hahahah, very good point. My library is only a few thousand photos and I haven't done much with it yet.

[mhdask]

This is odd. TypeORM seems to check if the table exists before actually trying to create it. typeorm/typeorm@e7649d2/src/migration/MigrationExecutor.ts#L496
Could this be an issue with the search path?

@Scrumplex Is this something i can check?

It would be interesting to see if there are any unexpected differwnces between the original database dump from the container and your current state. Perhaps you could try to dump your current NixOS database and diff the two files. Obviously the media paths are going to be different, but perhaps something else is wrong too?

Im not sure which two dumps you want to compare, the one from the original (docker) database dump, and the new nixos service version?
I actually dont have difference in media paths, but I do run into the problem with the earthdistance thing.

UPDATE:
I finally figured it out - In my old setup, my database user was called postgres, so i had to run a command:
sed "s/Owner: postgres/Owner: immich/g" immich-fixed-search-path.sql to fix that issue.
After running that, and importing the database, I no longer got that same error

[akho]

Thank you for the guide. Commented on two points which caused issues. I think we need someone to run this beginning-to-end one more time.

[akho]

Both points seem fixed, thank you.

I’ve already done things in my new module-based install, so it’s not easy to do a run-through the new version as-is...

[akho]

I've also had to re-create my external libraries, due to path mismatches with the container. Only noticed this now. Not sure what should have been done at db dump level.

This is really unfortunate, though — a lot of photos there…

[Atemu]

Shouldn't you only have to adjust the import paths of the existing libraries? You can do that in the UI without touching the db.

[akho]

Shouldn't you only have to adjust the import paths of the existing libraries? You can do that in the UI without touching the db.

Didn’t work for me.

[Scrumplex]

Should we merge this as-is? I think we could also move this into the NixOS wiki alternatively and allow people to refine it there.

[getchoo]

I think we could also move this into the NixOS wiki alternatively and allow people to refine it there.

I don't think that would be too bad of an idea. This is a pretty big section in the manual for something that's fairly specific

[Atemu]

I feel like it's worth having in the official channel-specific "proper" documentation rather than the wiki.

I've been meaning to have another look, try it and then merge but I've been putting off migrating my Immich setup...

[karb94]

I just completed my migration and I would like to share my experience following this guide.

1. The second sed command needs the -E flag to allow the use of parenthesis without escaping them
2. The second sed command did not replace multiple instances of 'upload/' in the same line. This is because it matches eagerly until the next forward slash ((.*)/) which in the case of multiple 'upload/' paths will match until the last one so the '/g' flag is useless. Instead, preventing the pattern from matching the tab character (used to separate the different fields) produced the desired result

sed -E "s#upload/([^\t]+)/#$TARGET_LOCATION/\1/#g" ./immich-fixed-search-path.sql > ./immich-final.sql

3. I don't understand why this guide assumes that your postgres database username (DB_USERNAME) is 'immich' in the docker image. The default value upstream is 'postgres' so this guide won't work for 90% of the people (including myself and @mhdask). The sed command provided by @mhdask to fix database username issue wasn't converting all instances of 'postgres' into 'immich' so I ended up just using sed 's/postgres/immich/g' which seems to work fine.