Fully defined apt repository package checking

[theofficialgman]

resolves 27f5fe1#r139018751

these modified and new functions fully define the checking of an apt repository for installed packages.

This prevents false positives that did occur before if multiple apt repositories shared a URI and SUITE but had different COMPONENTS

This also greatly improves support for .sources files where multiple stanzas are now properly supported and parsed

[theofficialgman]

just some examples where the output before is wrong

gman@gman-jam:~$ anything_installed_from_repo https://Pi-Apps-Coders.github.io/box64-debs/debian
gman@gman-jam:~$ echo $?
1
gman@gman-jam:~$ anything_installed_from_repo https://Pi-Apps-Coders.github.io/box64-debs/debian/
gman@gman-jam:~$ echo $?
1

no output, there should be debian-tegrax1

correct output now:

gman@gman-jam:~$ anything_installed_from_uri_suite_component https://Pi-Apps-Coders.github.io/box64-debs/debian/ ./
gman@gman-jam:~$ echo $?
0

0 is a package was found

[theofficialgman]

there were also false negatives. I had sublime text and sublime merge installed. I then uninstalled sublime text and it removed the shared apt repo while sublime merge apt package was still installed.

all these issues are fixed with this new function and its use

[Botspot]

With tr used here, it seems to me like the output from this would always be a single line, meaning all matching packages would be passed to apt-cache in one run.
So it seems like the only use for xargs is shorthand to avoid running apt-cache if no packages from the repo are installed?

[theofficialgman]

apt-cache policy needs the input as individual arguments. it also does not read from stdin. these are the reasons why xargs is used.

[Botspot]

I explained poorly.
If apt-cache is only being run once, then something like this should work if I understand this correctly

local apt_cache_policy_output="$(apt-cache policy $(echo "$packages_in_repo" | list_intersect "$installed_packages" | tr '\n' ' '))"

[theofficialgman]

No I correctly interpreted what you said.

apt-cache policy needs the input as individual arguments.

What you just sent puts all output as one input argument I think. Can't check now.

[Botspot]

No quotes around the inner $() will make each package into a separate argument.
If nothing else, this is simply something useful to know. I'm not necessarily requesting any changes here.

[Botspot]

I have changed the how the list of installed packages is determined, directly using the dpkg status file which is much faster.
This should be safe because on my system the original and new command return exactly the same output.

pi@raspberrypi:~ $ grep -xF 'Status: install ok installed' /var/lib/dpkg/status -B 2 | grep '^Package: ' | sed 's/^Package: //g' | sort | sha256sum
3ac304de04b498fe60eaaa74122744f08c5318c96eea28523fa860d16bfc40ca  -
pi@raspberrypi:~ $ apt list --installed 2>/dev/null | tail -n +2 | awk -F/ '{print $1}' | sort | sha256sum
3ac304de04b498fe60eaaa74122744f08c5318c96eea28523fa860d16bfc40ca  -

Time improvement: before=0m0.670s after=0m0.014s
After having trouble getting the outputs to match and finding out why, I realized that -B2 is necessary instead of -B1 for the handful of packages that add Essential: yes between the Package: line and the Status: install ok installed line. Meaning that our package_installed() function has been flawed this whole time.

[Botspot]

Also fixed setting installed_packages multiple times. Not sure why you had that inside the loop.

[theofficialgman]

Also fixed setting installed_packages multiple times. Not sure why you had that inside the loop.

that was just an oversight. no other reason

[theofficialgman]

After having trouble getting the outputs to match and finding out why, I realized that -B2 is necessary instead of -B1 for the handful of packages that add Essential: yes between the Package: line and the Status: install ok installed line. Meaning that our package_installed() function has been flawed this whole time.

Yup I discovered that yesterday as well before opening this PR. I intended to fix it in pi-apps but hadn't gotten around to it yet. I see you have already done so.

[theofficialgman]

I have changed the how the list of installed packages is determined, directly using the dpkg status file which is much faster. This should be safe because on my system the original and new command return exactly the same output.

I may change this back. The intention was to only get output for all and the current dpkg architecture. Currently any architecture package (including a non-native architecture package) will return. Pi-Apps package apps install the current dpkg architecture package exclusively (and that is by design) so it would be incorrect to mark one as "installed" if the user had a non-native architecture version of that package installed. Same issue with the package_installed() function.

In order to obtain the architecture information from dpkg, we would need to parse 8 lines for each package, instead of the current 2.

There is actually (at least one) case where the output from the grep command and the apt command differs once "correctly" sorted for architecture

using your dpkg command, it now becomes

grep -xE 'Architecture: ((arm64)|(all))$' /var/lib/dpkg/status -B 7 | grep -xF 'Status: install ok installed' -B 2 | grep '^Package: ' | sed 's/^Package: //g' | sort

using the apt-cache policy, it now becomes

apt list --installed 2>/dev/null | grep -E "/* * arm64|all " | awk -F/ '{print $1}' | sort

the one difference between the output is the dpkg command incorrectly filters out any manually held back packages while the apt command does not. I am currently holding back the qemu-user-static package on my main desktop linux install because of QEMU regressions that cause ARM64 segfaults https://gitlab.com/qemu-project/qemu/-/issues/1913. The output of such a held back package is Status: hold ok installed

to correct that and achieve the same result, now the command needs to be

grep -xE 'Architecture: ((arm64)|(all))$' /var/lib/dpkg/status -B 7 | grep -xE 'Status: ((install)|(hold)) ok installed$' -B 2 | grep '^Package: ' | sed 's/^Package: //g' | sort

now that is getting quite lengthy

I feel like the dpkg status grep, although faster and I knew that, might be prone to breakage in the future.

[Botspot]

I feel like the dpkg status grep, although faster and I knew that, might be prone to breakage in the future.

ok understand

[Botspot]

OK once you have uris sorted out I think this is ready to be merged.