Add CUDA/HIP support and GPU builds

[sebastiangrimberg]

Adds support for CUDA and HIP in the build system and makes corresponding changes to the code to run on GPUs.

Accompanying documentation update in #185

Resolves #3

TODO (for Spack support):

• Spack fixes are waiting on [obsolete by using @=main] Rename LIBXSMM main build to develop to avoid conflict with main-2023-11 spack/spack#42726 (Resolved by using libxsmm@=main dependency)
• We might be able to test CUDA and HIP builds in the CI (without a GPU/drivers) using Spack (Skip for now, since these take a while and require some system configuration to actually work)

[hughcars]

Generally looks good, mainly just clarification questions. The build process is clearly fiddly, so I wonder whether the documentation would be sufficient as is, but that can be tackled later.

The actual issues seen so far:

• Running the examples tests on a g5.48xlarge, the spheres case is failing with nans for the indicator norm.
• Running the examples with nt > 1, SuperLU seems to cause

  Residual norms for GMRES solve
 ** On entry to DGEMM  parameter number  8 had an illegal value
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 10743 on node c889f3baddb0 exited on
signal 11 (Segmentation fault: 11).
--------------------------------------------------------------------------

on the cpw_lumped_uniform case, whereas using STRUMPACK is fine. Might be as easy as noting in the docs that SuperLU doesn't play nice with threading right now.

[hughcars]

Nit: /= norm.

[sebastiangrimberg]

The actual issues seen so far:

• Running the examples tests on a g5.48xlarge, the spheres case is failing with nans for the indicator norm.

This is fixed in a8d0cc9.

• Running the examples with nt > 1, SuperLU seems to cause

  Residual norms for GMRES solve
 ** On entry to DGEMM  parameter number  8 had an illegal value
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 10743 on node c889f3baddb0 exited on
signal 11 (Segmentation fault: 11).
--------------------------------------------------------------------------

on the cpw_lumped_uniform case, whereas using STRUMPACK is fine.

As far as I can tell, this is only happening on M1 macOS with SuperLU_DIST. I'm strongly inclined to say it is a bug there but at least for now it is not related to this PR. One thing to probably explore is for OpenMP builds to build SuperLU_DIST and STRUMPACK without OpenMP and just rely on a threaded BLAS/LAPACK.

EDIT: SuperLU_DIST issue: xiaoyeli/superlu_dist#159

[sebastiangrimberg]

Once this PR, #193, and #194 are approved, I will rebase on main and merge all into main together.

[hughcars]

Approved as part of testing on #204