v0.11: Redesign config and build system, split into sub-crates, and provide disk image builder

[phil-opp]

This pull requests implements the next version of the bootloader crate: v0.11. It contains multiple breaking changes:

• Separate API crate: The bootloader is now split into two parts: An API crate to make kernels loadable by the bootloader and the actual bootloader implementation. This makes the build process for kernels much easier and faster.
• New config system: Instead of configuring the bootloader via a special table in the Cargo.toml, the configuration now happens through a normal Rust struct, which is part of the entry_point! macro. The macro then serializes the config struct at compile time and places it in a special ELF output section. The compile time serialization happens through a manually implemented const fn of the config struct.
• Load the kernel at runtime: The bootloader is now able to load files from FAT partitions at runtime. Thus, we don't need to link the kernel into the bootloader executable anymore. As a result, we don't need to recompile the bootloader on kernel changes anymore. We also load the config at runtime from the kernel's ELF section, which eliminates the second reason for recompiling the bootloader as well.
• Split into sub-crates: Since the bootloader build process does not need access to the kernel executable or its Cargo.toml anymore, we can build the different parts of the bootloader independently. For example, the BIOS boot sector is now a separate crate, and the UEFI bootloader is too.
• Library to create disk images: To create an abstraction the complex build steps of the different bootloader executables, we compile them inside cargo build scripts. At the top level, we provide a bootloader library crate, which compiles everything as part of its build script. This library includes functions for creating BIOS and UEFI disk images for a given kernel. These functions can be used e.g. from a builder crate or a build script of the downstream operating system.

Taken together, these changes should make the bootloader crate much easier to use and also more accessible to contributors.

---

This PR is still a work in progress. There are still many things missing, including:

• The BIOS implementation
  • Integrate into build system
  • Pure-Rust boot sector (16-bit real mode) that loads the second stage from a FAT partition MBR partition entry
  • Pure-Rust second stage (16-bit unreal mode) that initializes the CPU and loads the kernel and third stage from the FAT partition
    • Parse the FAT partition
    • Locate the kernel-x86_64 file in the root folder
    • Set up protected or unreal mode
    • Load kernel to protected mode memory
    • Create memory map and pass it to stage 3
    • Set up framebuffer
      • search VESA modes for a suitable one
      • fill in framebuffer info in boot information
      • enable the chosen VESA mode
      • consider config of kernel ELF (maybe later, not clear how we can read the config yet) -> postponed, tracked it BIOS booting: Consider framebuffer config of kernel ELF #281
    • Load third stage and jump to it
  • Stage 3 (protected mode)
    • Use framebuffer logger instead of VGA text mode
    • Set up long mode and identity mapping
      • Identity mapping
      • Compatibility mode
      • 64-bit mode
    • Jump to stage 4
  • Stage 4 (long mode)
    • link with the common crate to handle the config and map the kernel
• Adjust README for new version
• Create migration guides
  • From version 0.9.x
  • From version 0.10.x
• Clean up the repo and remove all legacy files
• For the kernel-requested physical memory mapping we don't want to skip regions. However, instead of mapping the full 0..max_phys_addr range, we could instead iterate over the reported regions and map only them. See better handling of very large memory maps #259 for details. -> we will fix this in a follow-up PR
• Fix BIOS boot error on real machine caused by VESA: v0.11: Redesign config and build system, split into sub-crates, and provide disk image builder #232 (comment)
• Improve error message when kernel has no bootloader-config section
• Try to find a way to keep the bootloader-config section when lto is enabled (see v0.11: Redesign config and build system, split into sub-crates, and provide disk image builder #232 (comment))

[phil-opp]

Turns out that there is a very similar (yet unpatched) bug that someone else found about three years ago: https://bugzilla.tianocore.org/show_bug.cgi?id=2372 and this is actually what went wrong when I was testing. I confirmed by attaching a debugger and reverse engineering the part of the firmware that crashed. This bug hasn't been patched yet, so, sadly, we still can't use custom memory types.

Thanks for looking into this! Even if this is fixed at some point, there will still be systems with older firmware in the wild, right? So we should probably just keep using the default memory type.

[Freax13]

Thanks for looking into this! Even if this is fixed at some point, there will still be systems with older firmware in the wild, right? So we should probably just keep using the default memory type.

Yup, I don't think we'll ever get to the point where all devices are patched. It would be nice not to rely on the workaround, it's really a shame that this bug exists, I liked the custom memory types 😞. I mainly looked into this because I couldn't believe that reputable vendors would push firmware that contained bugs that have been patched more than 10 years ago.

[Freax13]

When the alpha was released, docs.rs failed to build documentation for it: https://docs.rs/crate/bootloader/0.11.0-alpha.0/builds/539317. The build.rs script tried to build the uefi and bios bootloaders, but didn't have network access to download the dependencies and failed.

[phil-opp]

I think this is ready to be merged 🎉.