immich: init at 1.115.0; nixos/immich: init module #324127

jvanbruegge · 2024-07-02T19:22:48Z

Description of changes

This adds the package, nixos module and nixos test for immich

Closes #244803

Things done

Add a 👍 reaction to pull requests you find important.

h7x4 · 2024-07-02T19:26:17Z

Does this superseed #244803?

jvanbruegge · 2024-07-02T19:27:09Z

Yes, I was asked by @Atemu to open a new PR

eclairevoyant

Thank you greatly for your work on this! I had some feedback and questions below:

nixos/modules/services/web-apps/immich.nix

pkgs/by-name/im/immich/package.nix

nixos/modules/services/web-apps/immich.nix

pkgs/by-name/im/immich/package.nix

pkgs/top-level/all-packages.nix

pkgs/by-name/im/immich/package.nix

jpds · 2024-07-05T08:06:59Z

I occasionally see these messages in my logs:

server[36665]: [Nest] 36665  - 07/05/2024, 12:00:00 AM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-preview): Error: Cannot find ffprobe
server[36665]: [Nest] 36665  - 07/05/2024, 12:00:00 AM   ERROR [Microservices:JobService] Error: Cannot find ffprobe

server[8610]: [Nest] 8610  - 07/02/2024, 10:35:52 PM    WARN [Api:ServerInfoRepository~dfqq7ca6] Failed to read build-lock.json

Atemu · 2024-07-05T19:39:14Z

ffprobe is provided by the ffmpeg the build already depends on. Perhaps it needs to be wrapped to have access to the binary at runtime.

crertel · 2024-07-06T06:40:25Z

Just wanted to say I'm psyched to see this getting added. Thanks for the work!

nixos/modules/services/web-apps/immich.nix

jvanbruegge · 2024-07-06T11:20:12Z

I have addressed most of the review comments now and rebased onto the current master. This broke albumentations, which I fixed in a seperate commit.
We will still need to wait for the upstream PR to get the geonames without downloading the full docker container

jvanbruegge · 2024-09-23T23:06:15Z

Yes, I also upgraded my deployment to the latest version of this PR and everything works

dotlambda · 2024-09-24T00:00:26Z

pkgs/by-name/im/immich/update.sh

+done
+
+rm "$lock"
+cp "$sources_tmp" sources.json


Suggested change

cp "$sources_tmp" sources.json

mv "$sources_tmp" sources.json

if you don't want to leave temporary files behind

pkgs/by-name/im/immich/machine-learning.nix

dotlambda · 2024-09-24T00:05:47Z

pkgs/by-name/im/immich/machine-learning.nix

+    ]
+    ++ uvicorn.optional-dependencies.standard;
+
+  doCheck = false;


Can you add a comment with the reason?

Is there a problem with our opencv4 package?

AttributeError: module 'cv2' has no attribute 'Mat'

I enable tests in #344237.

dotlambda · 2024-09-24T00:06:31Z

pkgs/by-name/im/immich/machine-learning.nix

+  # Override Python packages using
+  # self: super: { pkg = super.pkg.overridePythonAttrs (oldAttrs: { ... }); }
+  # Applied after defaultOverrides
+  packageOverrides ? self: super: { },


Why would this be needed?

removed in #344237

dotlambda

After understanding the PYTHONPATH thing my remaining comments shouldn't block the merge.

jlbribeiro · 2024-09-24T13:03:39Z

Given the huge popularity of Immich, I'd like to take a bit of time to really thank all the people involved in packaging this.

@jvanbruegge I'd like to thank you for bringing this to the finish line; I know it probably wasn't easy, given this PR was open almost 3 months ago. You started as a user requesting a package and ended up stepping up and sharing your work with the rest of the NixOS community, when it looked like the consensus was having a separate project. I'm sure the community will help you maintain both the package and the module up to date, given the frenetic rate at which Immich is being developed :)

There are a couple of unsung heroes I'd like to mention:

@oddlama, for taking one of the first stabs at this more than a year ago!
@diogotcorreia, for having packaged one of the blockers, pgvecto.rs! You also started by sharing your work in a comment, and ended up making it available to the whole nixpkgs and unblocking @oddlama / @jvanbruegge efforts! Thank you!

And a huge thanks to all the reviewers involved!

Titaniumtown · 2024-09-24T14:29:17Z

Amazing work! How can I properly integrate this into my nix config? Does anyone have a public server config I can take a look at.
Thanks!

jvanbruegge · 2024-09-24T14:33:57Z

https://github.com/jvanbruegge/server-config/blob/86d8f6ff4c9e576cdfd94410dd4e35e58c6e8e5d/caladan/services.nix#L10

Titaniumtown · 2024-09-24T14:37:14Z

Thank you @jvanbruegge, just trying to figure out how to integrate this with the differing directories I have things stored in. Thanks again for all your hard work! Very much appreciated!

onny · 2024-09-24T15:09:53Z

back then i wrote this wiki page but not sure if its still up-to date https://wiki.nixos.org/wiki/Immich

ISibboI · 2024-09-24T15:25:59Z

Thank you so much for finally bringing immich to nixos!

I have one question: does the default setup support external libraries? Because I tried to add my local picture library on my server to immich as external library, but it cannot access any path I specify. It always gives EACCESS, even though the whole path is readable and executable by everyone.

oddlama · 2024-09-24T16:05:21Z

@ISibboI This is probably because of the service hardening I did in the first version. You will have to add your directory to a ReadWritePaths directive for the relevant systemd units to make it work

nixos/modules/services/web-apps/immich.nix

NyCodeGHG · 2024-09-24T17:25:53Z

@oddlama

This is probably because of the service hardening I did in the first version. You will have to add your directory to a ReadWritePaths directive for the relevant systemd units to make it work

This should be done automatically if it's not the default state directory imo.

jvanbruegge · 2024-09-24T17:31:54Z

I don't think that's the issue, as I also do not use the standard path and have no issues

PowerUser64 · 2024-09-24T18:31:55Z

Congratulations @jvanbruegge! This was one of the most through reviews I've seen, and you made it through! Thanks for all your hard work as well as the hard work of others to get this merged!

Scrumplex · 2024-09-24T19:46:49Z

I was able to migrate my Docker Compose deployment. An issue I ran into was that file paths in the database were relative to immich's working directory, meaning that everything was prefixed with upload/. I was able to remedy this by replacing upload/ with my mediaLocation (which is also what the UPLOAD_LOCATION variable from my Compose deployment pointed to).

Note

This is what worked for my installation that is a few years old. I am not sure if newer installations behave differently. Some facts about my Docker Compose installation:

The database is called immich
The database user is called immich
My UPLOAD_LOCATION (docker mount path) is set to /media/immich-library

Make a backup of your current library and save a copy of the following database dump
Enable NixOS Immich and switch. This will start an empty instance of Immich but will create the database and database user for us.¹ In my case it looked like this:

services.immich = {
  enable = true;
  mediaLocation = "/media/immich-library";
};

Stop NixOS Immich
- systemctl stop immich-server.service immich-machine-learning.service
Stop Docker Immich. Leave your postgres container running
Dump your database. The following command is going to dump it into Postgres' data directory, so it will be easy to access from your host system.
- docker compose exec database pg_dump --user immich -f /var/lib/postgresql/data/dump.sql immich
Fix unset search_path variable.²
- sed -i "s#SELECT pg_catalog.set_config('search_path', '', false);#SELECT pg_catalog.set_config('search_path', 'public', false);#" </path/to/immich/compose>/postgres/dump.sql
Replace upload/ path prefix.³ In my case I replace it with /media/immich-library.
- sed -i "s#upload/#/media/immich-library#g" </path/to/immich/compose>/postgres/dump.sql

Warning

In cases where a file is somehow stuck in your <library>/uploads/ folder, the previous sed command will create an invalid path (i.e. upload/upload/... -> /media/immich-library//media/immich-library/...). Make sure to catch these cases if present. In my case there was a single file that was referenced like this in the assets table and I was able to fix it manually after the fact.

Prepare NixOS database

sudo -u postgres psql immich
immich=# <paste following codeblock and exit>

DROP SCHEMA public CASCADE;
DROP SCHEMA vectors CASCADE;

CREATE SCHEMA public;

CREATE EXTENSION IF NOT EXISTS unaccent;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS vectors;
CREATE EXTENSION IF NOT EXISTS cube;
CREATE EXTENSION IF NOT EXISTS earthdistance;
CREATE EXTENSION IF NOT EXISTS pg_trgm;

ALTER SCHEMA public OWNER TO immich;
ALTER SCHEMA vectors OWNER TO immich;
GRANT SELECT ON TABLE pg_vector_index_stat TO immich;

ALTER EXTENSION vectors UPDATE;

Import your dump. Make sure the file is readable. I moved it to a location that is readable by the postgres user.
- sudo -u postgres psql immich -f </path/to/immich/compose>/postgres/dump.sql
Start NixOS Immich
- systemctl start immich-server.service immich-machine-learning.service
Make sure you can actually view images on the web app.

If any of these steps fail, you should have your original (unmodified!) database as well as a backup of your media location.

¹: As a precaution, you could make your existing Immich library read-only (i.e. chown -R root:root /media/immich-library) to avoid any modifications done by this empty installation of Immich.
²: Restoring a dump that uses Postgres earthdistance seems to fail if search_path isn't set to the target schema. See this StackOverflow comment
³: Perhaps you can inspect your database dump for this first. In my case every file reference started with upload/ (i.e. upload/thumbs/ instead of just thumbs/)

I hope this helps someone out there trying to get this working.

Atemu · 2024-09-24T20:16:10Z

@Scrumplex could you PR that into a migration doc and link it in the patch notes?

Titaniumtown · 2024-09-24T21:07:10Z

Thank you @Scrumplex, I've been working on migration now, luckily I had a backup of my sql because I've broken it hahaha

Scrumplex · 2024-09-24T21:38:59Z

I have written more in the last hour than I did on average per day for my bachelor's thesis. 😅

See #344300 for my proposed NixOS manual documentation.

It also includes a much better sed expression than I have commented here, that might be less prone to issues.

I'll keep "Allow edits and access to secrets by maintainers" on in case someone with commit access wants to commit changes into my PR. I will go to bed now! 💤

rhoriguchi · 2024-09-25T02:56:36Z

Found an issue with the module, when running immich and changing the group services.immich.group = "test"; the redis service will have the wrong group and fail to start. The issue is this line, so it would make sense to just create the group if the user is immich.

nixpkgs/nixos/modules/services/databases/redis.nix

Line 356 in 23cbb25

Group = conf.user;

> journalctl -r -u redis-immich.service
Sep 25 04:42:53 HOSTNAME systemd[1]: Failed to start Redis Server - redis-immich.
Sep 25 04:42:53 HOSTNAME systemd[1]: redis-immich.service: Failed with result 'exit-code'.
Sep 25 04:42:53 HOSTNAME systemd[1]: redis-immich.service: Control process exited, code=exited, status=216/GROUP
Sep 25 04:42:53 HOSTNAME (rep-conf)[13315]: redis-immich.service: Failed at step GROUP spawning /nix/store/338pjs37wgmhghid8wbai5qgyssihq2n-redis-immich-prep-conf: No such process
Sep 25 04:42:53 HOSTNAME (rep-conf)[13315]: redis-immich.service: Failed to determine group credentials: No such process
Sep 25 04:42:53 HOSTNAME systemd[1]: Starting Redis Server - redis-immich...

gnull · 2024-09-25T18:24:01Z

Anyone had success testing this on aarch64?

bct · 2024-09-26T13:58:19Z

If you use a non-socket postgres connection then services.immich.environment refers to cfg.database.port, but the module doesn't declare this option. Is this a bug, or am I missing something?

Titaniumtown · 2024-09-26T14:02:00Z

@bct i believe that is a bug, I can also reproduce.

jvanbruegge · 2024-09-26T14:06:53Z

Yes, that probably got lost somewhere during all of the rebasing for the reviews

aciceri · 2024-09-26T14:44:35Z

Anyone had success testing this on aarch64?

It tries to build python3Packages.insightface which is broken on aarch64-linux, don't know if it would work removing it (without face recognition obviously).

Moreover postgresqlPackages.pgvecto-rs too is broken on aarch64-linux and there may be even other broken dependencies.

Atemu · 2024-09-26T15:10:30Z

Sorry to interject but please create separate issues rather than commenting on this PR unless it is specifically something that requires the attention of the people who were involved in this PR.

A lot of people are subscribed and this code is in Nixpkgs now, so it should be treated like any other Nixpkgs code.

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation 8.has: changelog 8.has: module (update) This PR changes an existing module in `nixos/` labels Jul 2, 2024

jvanbruegge mentioned this pull request Jul 2, 2024

nixos/immich: init at 1.79.1 #244803

Closed

16 tasks

h7x4 added 8.has: module (new) This PR adds a module in `nixos/` 8.has: tests This PR has tests labels Jul 2, 2024

jvanbruegge force-pushed the immich branch from 185fa2a to b87f78c Compare July 2, 2024 19:36

jvanbruegge changed the title ~~nixos/immich: init at 1.107.0~~ nixos/immich: init at 1.107.1 Jul 2, 2024

eclairevoyant reviewed Jul 2, 2024

View reviewed changes

numinit reviewed Jul 2, 2024

View reviewed changes

nixos/modules/services/web-apps/immich.nix Outdated Show resolved Hide resolved

pkgs/by-name/im/immich/package.nix Show resolved Hide resolved

ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jul 2, 2024

eclairevoyant reviewed Jul 3, 2024

View reviewed changes

numinit reviewed Jul 6, 2024

View reviewed changes

nixos/modules/services/web-apps/immich.nix Outdated Show resolved Hide resolved

Aleksanaa changed the title ~~nixos/immich: init at 1.107.1~~ immich: init at 1.107.1; nixos/immich: init module Jul 6, 2024

jvanbruegge force-pushed the immich branch 5 times, most recently from 45c65e3 to 49d559c Compare July 6, 2024 11:18

github-actions bot added the 6.topic: python label Jul 6, 2024

mweinelt approved these changes Sep 23, 2024

View reviewed changes

dotlambda reviewed Sep 24, 2024

View reviewed changes

dotlambda approved these changes Sep 24, 2024

View reviewed changes

qbit approved these changes Sep 24, 2024

View reviewed changes

happysalada merged commit d026e3f into NixOS:master Sep 24, 2024
27 checks passed

jvanbruegge deleted the immich branch September 24, 2024 09:58

dotlambda reviewed Sep 24, 2024

View reviewed changes

nixos/modules/services/web-apps/immich.nix Show resolved Hide resolved

NixOS locked and limited conversation to collaborators Sep 26, 2024

	cp "$sources_tmp" sources.json
	mv "$sources_tmp" sources.json

immich: init at 1.115.0; nixos/immich: init module #324127

immich: init at 1.115.0; nixos/immich: init module #324127

Conversation

jvanbruegge commented Jul 2, 2024 • edited by eclairevoyant Loading

Description of changes

Things done

h7x4 commented Jul 2, 2024

jvanbruegge commented Jul 2, 2024 • edited Loading

eclairevoyant left a comment

Choose a reason for hiding this comment

jpds commented Jul 5, 2024

Atemu commented Jul 5, 2024

crertel commented Jul 6, 2024

jvanbruegge commented Jul 6, 2024

jvanbruegge commented Sep 23, 2024

dotlambda Sep 24, 2024

Choose a reason for hiding this comment

dotlambda Sep 24, 2024

Choose a reason for hiding this comment

dotlambda Sep 24, 2024

Choose a reason for hiding this comment

dotlambda Sep 24, 2024

Choose a reason for hiding this comment

dotlambda Sep 24, 2024

Choose a reason for hiding this comment

dotlambda Sep 24, 2024

Choose a reason for hiding this comment

dotlambda left a comment

Choose a reason for hiding this comment

jlbribeiro commented Sep 24, 2024

Titaniumtown commented Sep 24, 2024

jvanbruegge commented Sep 24, 2024

Titaniumtown commented Sep 24, 2024

onny commented Sep 24, 2024

ISibboI commented Sep 24, 2024

oddlama commented Sep 24, 2024

NyCodeGHG commented Sep 24, 2024

jvanbruegge commented Sep 24, 2024

PowerUser64 commented Sep 24, 2024

Scrumplex commented Sep 24, 2024 • edited Loading

Atemu commented Sep 24, 2024

Titaniumtown commented Sep 24, 2024

Scrumplex commented Sep 24, 2024 • edited Loading

rhoriguchi commented Sep 25, 2024 • edited Loading

gnull commented Sep 25, 2024

bct commented Sep 26, 2024

Titaniumtown commented Sep 26, 2024

jvanbruegge commented Sep 26, 2024

aciceri commented Sep 26, 2024 • edited Loading

Atemu commented Sep 26, 2024

jvanbruegge commented Jul 2, 2024 •

edited by eclairevoyant

Loading

jvanbruegge commented Jul 2, 2024 •

edited

Loading

Scrumplex commented Sep 24, 2024 •

edited

Loading

Scrumplex commented Sep 24, 2024 •

edited

Loading

rhoriguchi commented Sep 25, 2024 •

edited

Loading

aciceri commented Sep 26, 2024 •

edited

Loading