Adds WARC generation for offline viewing

[gjvnq]

I added a gulp task to generate WARC files from the website so it can be read offline. These can viewed through WARC players like replayweb.page and they are attached to this pull request to allow the mantainers to evaluate this feature.

There's a minor bug about the sidebar/bookmarks which I'm still investigating (see webrecorder/replayweb.page#99).

Finally, WARCs aren't as good as EPUBs but they are here and don't have as many compatibility quircks as EPUBs.

---

Attached files: (you will likely need to rename the extensions from .warc.txt to .warc)

gdb-en.warc.txt
gdb-zh.warc.txt
gdb-de.warc.txt
gdb-hu.warc.txt
gdb-pl.warc.txt
gdb-zh.warc.txt
gdb-es.warc.txt
gdb-fr.warc.txt

[Twipped]

I've never heard of this format. Is there a meta tag or something that exposes it as available?

[gjvnq]

I don't understand your question. If you are asking about how the user would get the file, the answer is simple: through a traditional <a> link just like many websites use for PDF and EPUB. Em qui., 26 de mai. de 2022 11:58, Jocelyn Badgley ***@***.***> escreveu:

…

I've never heard of this format. Is there a meta tag or something that exposes it as available? — Reply to this email directly, view it on GitHub <#95 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXIYHFDITPJWT5QWFQ2J3LVL6GRHANCNFSM5W7PNBSQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Twipped]

Ah, okay, I thought it was something that browsers hooked into for offline caching.

So, one small problem with this is that I'm actually trying to get off of gulp. There's also several other changes that I just can't merge at this time without seeing what they do. I'll have to review this more in-depth later.

[gjvnq]

So, one small problem with this is that I'm actually trying to get off of gulp. There's also several other changes that I just can't merge at this time without seeing what they do. I'll have to review this more in-depth later.

I dislike gulp too. What do plan to use as a replacement?

And what can I use for WARC generation? Is it okay if I use something written in Go, Rust or Python?

[gjvnq]

Ah, okay, I thought it was something that browsers hooked into for offline caching.

I can use Manifests (see http://diveintohtml5.info/offline.html) to allow for offline browsing but I think that the usability is subpar.

[Twipped]

What do plan to use as a replacement?

Just plain JS tied together with yargs. The main thing holding me back right now is the AWS publish system. I haven't found anything else that supports pattern based cache control headers. I need to port gulp-awspublish out into a standalone library, but haven't the time to do it.

[Twipped]

Is it okay if I use something written in Go, Rust or Python?

Only if it's something quick and easy to install. Python dependency management is a hell I'd rather not get involved with, and I have zero experience with Go or Rust. I'd much rather it be in JS.

[gjvnq]

Is it okay if I use something written in Go, Rust or Python?

Only if it's something quick and easy to install. Python dependency management is a hell I'd rather not get involved with, and I have zero experience with Go or Rust. I'd much rather it be in JS.

Golang generates static binaries so installing them is pretty easy.

As for the current JS script (offline.js), it uses Gulp very minimally so converting it to work with another tool should be very easy.