cronjobs: inject canonical URLs into older manual pages (SEO)

[neteler]

The GRASS GIS manual pages of the different versions have been published for a long time with a difficult to understand concept of being invisible, redirected or shown, which also strongly affects the search engine ranking.

SEO: Without indication of "canonical" URLs different versions wipe each out out in search engines. Canonical tags help consolidate duplicate or similar content by specifying the preferred version of a page, ensuring search engines index and rank the desired URL while avoiding duplicate content issues.

This PR changes the cronjob scripts to

• inject "grass-stable" as the "canonical" into older manual pages under versioned URL
• inject "grass-devel" as the "canonical" into the development manual pages under versioned URL

Like this no "duplicate content" from a SEO perspective should occur.

Also robots.txt is updated to reactivate the manual pages of old GRASS GIS versions (which now contain "grass-stable" as the canonical).

Fixes OSGeo/grass#4579

[neteler]

Note: these files are now deployed on grass.osgeo.org for testing.

[echoix]

Wouldn't the grass-devel and grass-stable primary content be rather similar, thus being potentially penalized as duplicates?

How does this method handle pages that don't exist in later versions, or are renamed/moved?
It doesn't happen often, but we had some every now and then.

[neteler]

Wouldn't the grass-devel and grass-stable primary content be rather similar, thus being potentially penalized as duplicates?

Very good point.
I have changed it in 9824486 to inject in the 8.5 versioned and the "grass-devel" manual sections the "canonical" to point to "stable" rather than to "devel".

Deployed update on grass.osgeo.org, triggered cronjob and told Google Search about it.

How does this method handle pages that don't exist in later versions, or are renamed/moved? It doesn't happen often, but we had some every now and then.

A few of them are handled with redirects in Apache. I would not know any other method.

[neteler]

Too bad, now the building with cron_grass_preview_build_binaries.sh on the Debian grass.osgeo.org server is broken after the MD merge:

...
Parsing <v.what.strds.timestamp>... SUCCESS
Parsing <wx.metadata>... FAILED
Parsing <wx.mwprecip>... FAILED
Parsing <wx.stream>... FAILED
Parsing <wx.wms>... FAILED
+ cp /home/neteler/.grass8/addons/modules.xml /var/www/code_and_data/addons/grass8/modules.xml
+ export ARCH
+ export ARCH_DISTDIR=/home/neteler/src//main/dist.x86_64-pc-linux-gnu
+ export GISBASE=/home/neteler/src//main/dist.x86_64-pc-linux-gnu
+ export VERSION_NUMBER=8.5
+ python3 /home/neteler/src//main/man/build_keywords.py /var/www/code_and_data/grass85/manuals/ /var/www/code_and_data/grass85/manuals/addons/
Traceback (most recent call last):
  File "/home/neteler/src//main/man/build_keywords.py", line 202, in <module>
    build_keywords("md")
  File "/home/neteler/src//main/man/build_keywords.py", line 68, in build_keywords
    from build_md import (
  File "/home/neteler/src/main/man/build_md.py", line 264, in <module>
    man_dir = os.path.join(os.environ["MDDIR"], "source")
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'MDDIR'

Can @landam help?

[neteler]

Too bad, now the building with cron_grass_preview_build_binaries.sh on the Debian grass.osgeo.org server is broken after the MD merge:

Bugfix PR: OSGeo/grass#4739