Native AppStream support

[cbbayburt]

This RFC proposes native support for AppStreams repositories in Uyuni.

See the rendered version.

[agraul]

Can you expand on "as a single unit"? That sounds a bit like a transaction (succeed together or fail together), which we don't have in reposync.

The PoC script, as far as I can tell, works on already synced, imported, and linked packages. Is that correct? What are the downsides of doing it this way vs. doing it at a different time in the process?

[cbbayburt]

The new data we collect at this step is critical for modular repositories to work. If we somehow miss this data, Uyuni won't know if a package is really modular or not, and will treat it as a regular package. As a result, Uyuni will suggest wrong updates to existing packages (the initial problem we have with modularity).

Because of this, we have to make sure that module metadata is properly processed when syncing a repository. If not, the reposync for this channel should fail completely.

How do we currently make sure that the repos don't end up in an invalid state during reposync?

[cbbayburt]

How do we currently make sure that the repos don't end up in an invalid state during reposync?

It seems we do it using proper exception handling and DB transactions.

[mcalmer]

Maybe the module.yaml should be parsed first. I think during package import we call multiple times "commit" to solve problems with dependencies between packages which gets imported in parallel.

[cbbayburt]

We have to do it in the end because we need package IDs for the relation.

[rjmateus]

Shouldn't this be part of the change channels? Why we need a new place to define the assigned channels to a minions?
We should allow change to any, and auto-detect if the channels have modules enabled, should the "alternative" path

[cbbayburt]

Every modular repository has its own set of modules and theoretically, a system can have more than one modular channel assigned. In such cases, the user must first pick a channel to see its modules.

But after writing this, I think we don't need this step. Instead, we can put together all the modules from all the assigned modular channels and present them in a single list.

I'll update this part accordingly.

[rjmateus]

This means we would not be able to filter packages and is version inside a module? Would not make sense to have the module available if at least one package from a module is present?
This looks to be to restrictive to me.

[cbbayburt]

In most cases, a module has multiple packages packed together to form an application. For example:

Packages in the Postgresql module:

• pg_repack
• pgaudit
• postgres-decoderbufs
• postgresql
• postgresql-contrib
• postgresql-docs
• postgresql-plperl
• postgresql-plpython3
• postgresql-server
• ...

so it usually makes no sense to have a module enabled if not all the packages are there.

OTOH, different versions of a module are separate module entries, so we can still implement filters to selectively filter different versions (using the module versions notation).

[rjmateus]

I was thinking of scenarios where some packages in the module could be optional, and the user decided to exclude them in a filter. In this scenario, the module would never get available. Do you think a scenario like this one makes sense?

[cbbayburt]

Yes, that's done by "Profiles" in AppStreams. In the example above, postgresql module usually has at least 2 profiles, one named client and one named server. Depending on which profile is activated, the list of actual installed packages changes. But this needs careful consideration because the user might want to enable different profiles for different clients from the same repository.

So maybe the best way is to give the user the freedom indeed. If dnf fails because of some missing package, then it's up to them to fix it in CLM.

[mcalmer]

Possible problem: what happens if a customer want to allow only security updates and apply a filter like this?
Such an update do not match all all packages in a steam. We need to check what prevents to remove the whole stream incl. the security update.

[cbbayburt]

I also need to check how DNF behaves if some packages are missing in a module. Our strategy is to clone DNF's behavior as much as possible.

[Etheryte]

While I know this is in the unresolved questions section, are we roughly sure we can migrate everything we currently support in a reasonable way?

[cbbayburt]

The new approach is fundamentally different so we won't be able to offer an automated migration at all. So what I mean here is the migration of the customer's workflow.

We just need to make sure that the customers have a smooth way of switching without any disruptions with proper documentation. But in the end, the change is really just getting rid of many extra steps to achieve the same goal, so I'm sure the customers would be happy to go through the labor of migration.

[admd]

While I acknowledge that customers may welcome the transition to the new approach, we cannot compel them to do so by breaking stuff; it should be done at their convenience. In other words, we must ensure that their existing clm project setups remain functional.

Moreover, if it helps, we could consider applying this change solely to new CLM projects, potentially easing concerns regarding existing setups.

[cbbayburt]

Sure, it's a bit tricky because we need to replace the old AppStream filters with the new functionality, but at least we can implement it so their existing projects won't be affected until they want to rebuild them.

[rjmateus]

Would it be possible to implement it in a way that allows uses to continue using the flatten method and this one if they want?
Some customers may be familiar with the old one and may want to continue using it. Abid do you think such a case could exist?

[cbbayburt]

Flattening is only possible with the old-style AppStream filters. If we include both old and the new filters, it would be too confusing for users, and also they will keep using the old way for a longer time. That means more L3s and harder maintenance for us so I wouldn't recommend it.

[rjmateus]

Got it, and sounds good to me. 👍

[admd]

Can, I'm not fully informed on this topic. So please excuse my ignorance.

Could you clarify how we currently installing/removing/updating packages—are we using yum or dnf ?

If we're using yum, will we stick with it for package operations, or are we considering transitioning to DNF?

[cbbayburt]

Nowadays we're using the terms yum and dnf interchangibly, but since RH8, yum is just a wrapper around dnf. We still have a lot of mentions of yum in our python modules, but Salt is actually using dnf.

[admd]

thank you! So it's safe to say that for RHEL and clones, only DNF will be used at the end. Thank you Can.

[admd]

Again, I lack knowledge here so another question. What kind of dependency issue would you expect? Do streams depend on other streams or are they self-contained?

Ideally, we should have mechanism in place, so user doesn't get into bad state. Just like we do with required/recommended channels. Select all the needed stuff automagically when user select a module stream with option that experienced user can de-select.

But if that's asking for too much, this optional would be super helpful.

[cbbayburt]

Yes, a module may depend on other modules and they must be enabled together. If we miss this, dnf would complain about the missing dependencies. Other than that, mgr-libmod also checks for conflicting streams. If, for example the user tries to enable two streams of the same module, mgr-libmod would complain about this too.

[cbbayburt]

Btw, in real world, dependency relations so far only exist in RH8. RH9 has a lot less modules than 8, and we haven't seen any dependent modules in 9 yet. RH8 is kind of an edge case of modularity 😂

[mcalmer]

There is a 4th point needed. You need to generate new metadata for the Channels including new compiled modules.yaml from the database informations.

[mcalmer]

Maybe the module.yaml should be parsed first. I think during package import we call multiple times "commit" to solve problems with dependencies between packages which gets imported in parallel.

[mcalmer]

Possible problem: what happens if a customer want to allow only security updates and apply a filter like this?
Such an update do not match all all packages in a steam. We need to check what prevents to remove the whole stream incl. the security update.