Improving maintainability

[eimrek]

This repository (along with https://github.com/materialscloud-org/discover-pyrene-mofs and to a lesser degree also https://github.com/materialscloud-org/discover-hcofs-co2) is a bit difficult to maintain. Every time I update the aiida version in Materials Cloud, I need to update it also here, as the same aiida storage backend is used and it often updates to a new version. Additionally sometimes things break (e.g. conflict with graphviz version w.r.t. to aiida-core, see #32 and #33).

Any ideas how to improve this maintainability? @ltalirz @danieleongari @giovannipizzi

[ltalirz]

If the EXPLORE section accompanying these DISCOVER sections could be reading data from a REST API running inside the application container, you might be able to keep the aiida versions frozen from then onwards (to the degree that the materials cloud explore JS applications supports older REST APIs).

[eimrek]

thanks @ltalirz. I don't fully understand what you mean. The EXPLORE section works well and is easy to maintain (I just update the aiida version on my Materials Cloud servers and migrate all the aiida profiles, including the curated-cofs one).

Regarding this DISCOVER section, however, if i understand correctly, it mounts the aiida file repository and accesses the psql server of the Materials Cloud aiida instance and is directly running AiiDA queries on that. Couldn't this, in principle, access the data via the aiida REST API? Although this seems like it could be a bit of work...

[ltalirz]

thanks @ltalirz. I don't fully understand what you mean. The EXPLORE section works well and is easy to maintain (I just update the aiida version on my Materials Cloud servers and migrate all the aiida profiles, including the curated-cofs one).

Sorry, let me clarify: yes, migrating the explore sections is straightforward, but it is also what potentially breaks compatibility with AiiDA-based discover sections.

Therefore, a possible alternative is to keep the AiiDA database frozen in time and have he explore section be served from an AiiDA REST API running inside (either the same or a separate) container of the discover section.

Not saying this should be done, it's an option for reducing the maintenance burden (as long as the explore JS application remains backwards compatible to the AiiDA REST API used in the section).

Overall, I think it is also fair to look at the amount of traffic the discover sections get over time and use it as a signal to decide for how long to maintain them.
E.g. the curated cofs one I know is still being updated; the other two I don't think so.
The underlying data will remain available on the archive in any case.

[eimrek]

There was some discussion on this with @mpougin, who is interested in keeping this section online on materials cloud.

Also, thanks @ltalirz for the comments, what we discussed, matches with what you said.

If the app can be fully container in a docker container (without mounting files of the mcloud aiida instance or accessing the postgreSQL), then the maintenance burden is minimized. It could also be hosted on our dokku server.

So what i would propose is:

• directly copy/download the .aiida file in the docker-container.
• import it to the aiida instance
  • Note: potentially makes sense to use the new sqlite_zip readonly storage introduced in aiida-core 2.5.0.
    • verdi profile setup core.sqlite_dos -n --profile <PROFILE_NAME> --filepath <ARCHIVE>
    • This would e.g. bypass the need to install postgresql.

I think this is not too much work.