KindleGen is no longer available for download

[slonopotamus]

1. https://www.amazon.com/gp/feature.html?docId=1000765211 says "KindleGen is no longer available for download."
2. All old download links from http://kindlegen.s3.amazonaws.com now give access denied

See #363 (comment) for possible workarounds.

[slonopotamus]

Upstream issue: tdtds/kindlegen#42, though I doubt anything can be done here.

[abelsromero]

I have opened a support case with Amazon to see if something can be done. Not much faith, but who knows...

[abelsromero]

After some calls and couple of mails I just got a dismissive "thanks for your feedback and using Amazon products. We will pass those suggerences to the development team". 🤷‍♂️
I am afaid we are quite out of options.

[slonopotamus]

Thanks for trying.

[slonopotamus]

Let's make it clear.

Current workaround for Windows and MacOS is to:

1. Install Kindle Previewer
2. Locate KindleGen executable in it.
   For Windows, default location is %LOCALAPPDATA%\Amazon\Kindle Previewer 3\lib\fc\bin\kindlegen.exe
   For MacOS, default location is /Applications/Kindle Previewer 3.app/Contents/lib/fc/bin/kindlegen
3. Do one of
   a. Add that directory to PATH
   b. Set KINDLEGEN environment variable with path to KindleGen executable
   c. Use -a ebook-kindlegen-path=/path/to/kindlegen to command line when converting to Mobi

Current workaround for Linux is:

1. Find kindlegen_linux_2.6_i386_v2_9.tar.gz somewhere in the internets. Note that this step is legally questionable.
2. Unpack it
3. Do one of
   a. Add unpacked directory to PATH
   b. Set KINDLEGEN environment variable with path to KindleGen executable
   c. Use -a ebook-kindlegen-path=/path/to/kindlegen to command line when converting to Mobi

[slonopotamus]

Meanwhile, I've started development of a Ruby library that will (hopefully) be capable of reading/writing Mobi files: https://github.com/slonopotamus/dyck

It isn't clear how much time and effort it will require yet.

If this library ends up in something usable, Asciidoctor EPUB3 will use it to produce Mobi files.

[slonopotamus]

A bit of a status report: Dyck is currently capable of producing MOBI6 files that properly work in Calibre/Okular/Kindle Previewer. It's still missing some bits that allow KF8 to render, I'm working on that.

[abelsromero]

I've been wanting to ask how are you doing this because I assume there's no official documentation. And I just saw a comment in gitter.

Is that it? Checking other implementations?

[slonopotamus]

Checking other implementations?

Yep. There are at least three incomplete (to a certain degree) implementations I am aware of: Calibre, libmobi and mobi.rb. Both libmobi and mobi.rb mostly do reading, not writing. And Calibre is too much to depend on.

There's also (again, incomplete) reverse-engineered format documentation.

Combining all these together + some trial and error allows me to move forward :)

A bit of current status: I already have proper data structures to read KF8 produced by KindleGen, now need some code to write these structures back to files. Index records are pain.

[jcdarwin]

A possibly better option: use a dockerised version of kindlegen: https://github.com/koenrh/docker-kindlegen

This also works fine on 64 bit OSes such as Mac Catalina, which kindlegen can't run on natively (as it's only 32 bit).

[abelsromero]

@jcdarwin technically yes, but I have legal concerns. The kindlegen binary is propietary software, so only the owner (Amazon) should provide binary distributions.

[slonopotamus]

@jcdarwin If it was legally OK to redistribute KindleGen, it would simply be bundled in kindlegen gem (see tdtds/kindlegen#43). But KindleGen EULA doesn't allow redistribution.

[craigscott-crascit]

I'm wondering if we actually need to convert from the epub format to mobi any more. I was just experimenting with Kindle Previewer 3.49 (latest available release) and it no longer loads the previously generated .mobi files (it did previously for the same file(s)). However, it does load the *-kf8.epub file that asciidoctor-epub3 generates on its way to producing the .mobi files (when there is a kindlegen available). Furthermore, if you import a *-kf8.epub file into Kindle Previewer, it does so successfully and even enables Enhanced Typesetting. More interesting though, if you then try to export the result, you can choose *.kfp format (which seems to be specifically for reloading back into Kindle Previewer) or *.mobi format which it labels as "to side-load on older Kindle Devices". Selecting the latter, it pops up a dialog with the following note:

NOTE: Please use your original source file (EPUB or DOCX) for publishing. This MOBI file should only be used for testing on older devices that do not support Enhanced Typesetting.

This would suggest that epub is now the format that should be used even for Kindle. I'm wondering if the most appropriate format for distribution to Kindle is now the *-kf8.epub (i.e. still epub, just with tweaks for Kindle)? Can anyone confirm? If this is indeed the case, would it be possible to add an option to asciidoctor-epub3 to produce just that file and not try to run kindlegen on it? That is currently the missing piece for me to still be able to produce an output format for kindle through CI builds.

[craigscott-crascit]

Although it seems that Kindle Previewer screws up SVG images in documents it imports (loses text from the SVGs). The .mobi produced by asciidoctor-epub3 using the kindlegen that comes with Kindle Previewer doesn't have this problem. So yeah, maybe still need .mobi generation after all for now.

[slonopotamus]

All this terminology is kinda confusing, let's make it clear:

1. All .mobi/.azw files are actually PalmDB containers. Think of it as something like an archive that can hold multiple (this is important) entries inside. On this level of abstraction, file doesn't have any relation to ebooks at all.
2. There's MOBI6 ebook format. MOBI6 files are commonly named .mobi.
3. There's KF8 ebook format. To make things confusing, KF8 files can be named .azw, .azw3 or .mobi. One file can contain KF8 and MOBI6 at the same time and this is what kindlegen produces.
4. There's KF10 ebook format. Files are named .kfp. This is what Amazon calls "Enhanced Typesetting" thing. I don't have a prooflink currently, but it is very likely that a single file can contain all three MOBI6 + KF8 + KF10. UPD: Google says that KPF is only an intermediate format that is never uploaded to ebook. Instead, it is later converted to KFX.

So. Kindlegen is a EPUB3 -> MOBI6+KF8 converter.

Kindle Previewer also can do EPUB3 -> KF10 (supposedly, combined with MOBI6/KF8, I didn't check).

Now, what does "this file doesn't support Enhanced Typesetting" mean? It means that file is not a KF10 and not a EPUB3, so Kindle Previewer cannot convert it to KF10. Kindle Previewer doesn't have KF8 -> KF10 converter.

Problems with Kindlegen:

1. It no longer has legal distribution way.
2. It doesn't exist for modern MacOS, neither x64 nor ARM.
3. It is closed-source, so when it has bugs, they have to be hacked around in EPUB3 that is fed into Kindlegen (that's why asciidoctor-epub3 produces a bit different -kf8.epub file)
4. It cannot produce KF10, that's why Kindle Previewer complains about lack of Enhanced Typesetting.

Problems with Kindle Previewer:

1. It doesn't exist for Linux
2. It is also closed-source, so cannot be fixed directly

As I already said above, I've started writing a library that aims to be EPUB3 -> MOBI6 + KF8 converter. It already can do MOBI6 and almost can do KF8. I had little spare time last year to finish KF8 part, that will hopefully happen early this year. When it's done, asciidoctor-epub3 will be to produce KF8 files without any Kindlegen. After that, we could possibly work on adding KF10 support, though it might be harder because KF10 was not reverse-engineered/documented yet (or I just didn't find that info yet).

The overall goal is to be able to

1. Control resulting file, what is impossible with closed-source tools
2. Produce files on any platform, what is impossible with Kindle Previewer

[slonopotamus]

it no longer loads the previously generated .mobi files

Interesting, I didn't notice this when was reading your comment innitially. It is true, Amazon dropped KF8 support from Kindle Previewer:

http://kindlepreviewer.s3.amazonaws.com/UG_ReleaseNotes_EN.txt

New in Kindle Previewer 3.49.0:

1. Opening AZW/AZW3 files are not supported. We recommend using EPUB format for preview and publication instead.

[craigscott-crascit]

Thanks for that. There's also this just a little further down in the release notes too:

New in Kindle Previewer 3.48.0:
...
3) Kindle Previewer no longer supports opening MOBI files created from legacy versions of KindleGen or third party applications. We recommend using an EPUB file when creating a reflowable book.

So it basically seems like Kindle Previewer now only opens the latest KF10 format.

[gregturn]

Thanks @slonopotamus and team for all your efforts.

I, too, am stymied at how this entire process appears to have been forced to pivot in recent months thanks to slim information from Amazon.

I am in the midst of trying to roll out my latest tech book, and am getting confusing stuff from Kindle Previewer 3 (3.50.0). I tried opening the .epub (not -kf8.epub) version of things.

Two different pages, that appear structurally identical get rendered VERY differently by Kindle Previewer 3.

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xml+xhtml; charset=UTF-8"/>
<title>Foreword</title>
<link rel="stylesheet" type="text/css" href="styles/epub3.css"/>
<link rel="stylesheet" type="text/css" href="styles/epub3-css3-only.css" media="(min-device-width: 0px)"/>
</head>
<body>
<section class="chapter" title="Foreword" epub:type="chapter" id="hacking-with-spring-boot-classic-foreword">
<header>
<div class="chapter-header">
<h1 class="chapter-title"><small class="subtitle"><b>Foreword</b></small></h1>
</div>
</header>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis quis fringilla ex. Curabitur ornare ac lorem ac ullamcorper. Suspendisse blandit accumsan risus a consequat. Quisque sagittis cursus lacus. Curabitur fermentum volutpat malesuada. Aliquam sagittis ex at dolor tincidunt, eu ultrices nisl iaculis. Donec cursus, metus et aliquam pulvinar, mauris quam dignissim dolor, ullamcorper tristique leo elit quis augue. Etiam ultricies ante vitae enim molestie eleifend. Maecenas volutpat ante et ligula ornare mattis. Nam id luctus felis, at faucibus mi. Suspendisse eget erat sem. Aliquam orci elit, vestibulum eu aliquet non, commodo ut urna. Quisque fermentum magna sit amet eros interdum egestas. Mauris ullamcorper rhoncus neque, at consectetur turpis sollicitudin et. Duis semper quam felis, eu imperdiet sapien ultrices non.</p>
<p>Etiam quam dolor, gravida ac tincidunt nec, facilisis eu nunc. Maecenas porta fringilla metus, vitae finibus enim blandit vel. Pellentesque cursus orci eu eros fermentum, non pellentesque leo suscipit. Vivamus eleifend neque nec interdum finibus. Nullam in ante vel urna gravida venenatis at eget augue. Sed tempor quis arcu sed tempus. Suspendisse in felis ultricies, ornare mi et, tempor tortor. Pellentesque quis quam risus. Duis laoreet convallis mollis.</p>
<p>Phasellus pretium ultricies eros quis lacinia. Cras tempus aliquet tellus, id lacinia enim rutrum ac. Nunc consectetur at tortor quis lobortis. Donec dui mauris, venenatis sit amet tellus ac, tempor dictum lectus. Ut dui mauris, tempus et maximus et, blandit non risus. Donec sed lacus non ante ultrices maximus in vel metus. Phasellus nisi lorem, porttitor eu massa sit amet, vehicula dictum lectus. Phasellus ornare felis massa. Nam eu iaculis libero.</p>
<p>&#8201;&#8212;&#8201;<strong>Dr. Dave Syer</strong><br/>
<em>Senior Consulting Engineer and co-founder of Spring Boot</em></p>
</section>
</body>
</html>

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xml+xhtml; charset=UTF-8"/>
<title>About the Author</title>
<link rel="stylesheet" type="text/css" href="styles/epub3.css"/>
<link rel="stylesheet" type="text/css" href="styles/epub3-css3-only.css" media="(min-device-width: 0px)"/>
</head>
<body>
<section class="chapter" title="About the Author" epub:type="chapter" id="hacking-with-spring-boot-classic-about">
<header>
<div class="chapter-header">
<h1 class="chapter-title"><small class="subtitle"><b>About</b> <b>the</b> <b>Author</b></small></h1>
</div>
</header>
<p><strong>Greg L. Turnquist</strong> works on the Spring team as a principal developer at VMware. He is a committer to Spring HATEOAS, Spring Data, Spring Boot, R2DBC, and Spring Session for MongoDB. He wrote <em>Hacking with Spring Boot 2.3: Reactive Edition</em> as well as Packt&#8217;s best-selling title, <em>Learning Spring Boot 2.0 2nd Edition</em>. He co-founded the Nashville Java User Group in 2010 and hasn&#8217;t met a Java app (yet) that he doesn&#8217;t like.</p>
<p>Be sure to subscribe to <em>Spring Boot Learning</em>, the YouTube channel where you learn about Spring Boot and having fun doing it at <strong>YouTube.com/GregTurnquist</strong>.</p>
</section>
</body>
</html>

I have opened a ticket with KDP tech support and am attempting to correspond with them about what's happening, and will gladly share any details I get back.

At the same time, I'm happy to provide details to you if that would help resolve some of these issues.

If there a better ticket (or a new ticket) to comment on, I'd be happy to contribute there as well. And if you need a reproducible build, I can generate that as well, if it's constructive to your coding process.

[HonkingGoose]

@slonopotamus This can be unpinned I guess, as this is now resolved?

[slonopotamus]

Yep, done

[vuchien000]

ilius/pyglossary#299