Feature Request: Pretty-print SVG file and group elements with nested inherited properties to reduce file size

[cookiengineer]

It would be amazing to have a pretty-printed variant of the SVG file when it is initially rendered. By pretty-printed I mean all <text> sections at least line-separated.

Why? That way it would be possible to track the SVG animations via git! Imagine the possibilities ... when new releases come out, you could even see the differences in the SVG animation files.

[trevordmiller]

Prettier might be helpful for this:
https://prettier.io

[nnist]

I created a pull request which should fix this issue, it should nicely format the resulting SVGs.

@trevordmiller
I just tried formatting a rendered .svg using prettier, but it gives a syntax error no matter which parser I use.
It looks like XML/HTML/SVG isn't supported yet: prettier/prettier#3491

[cookiengineer]

@trevordmiller You actually won't need prettier for this, as you could just use xmllint for it locally. The underlying issue is that the SVG file is rendered in an explicit and redundant way, not using groups and not inheriting most of the properties of the file itself. You can reduce the file size by deca-percentages literally by just deleting the unnecessary properties on each <text> entry, especially when you have curl with a progress bar running somewhere, because it wastes hundreds and thousands of text nodes just because of the reset character (\r).

My concern and this issue was mostly about a pretty printing functionality, so that in step 2 somebody can trace down the properties that are unnecessary and can be grouped. Currently it's just a big mess.

If you want an example: I could reduce the file size of the installation svg of lycheejs from around 8 MB (megabyte!) to around 362kB with this, and a lot of manual editing macros in VIM.

[trevordmiller]

Ah good points :)

[nbedos]

You can reduce the file size by deca-percentages literally by just deleting the unnecessary properties on each entry, especially when you have curl with a progress bar running somewhere, because it wastes hundreds and thousands of text nodes just because of the reset character (\r).

@cookiengineer Would you mind recording a terminal session with curl in a cast file (termtosvg record) and sharing it here ? I tried reproducing the issue you're experiencing but I get a 154kB SVG for a curl session which lasted a minute and a half.

[cookiengineer]

@nbedos I don't have the original asciinema cast file left (recorded with termtosvg), but I have the fixed one. I'm trying to make a new one once I have a better internet connection (as it will literally take more than 60mins to install). I tried to make a reduced test case, as described below.

---

The original cast file had two types of issues that appeared with a slow internet connection (slow meaning 3-5MB/s and large repos/downloads beyond 1200MB+):

• git clone resulted in thousands of lines, also with mixed [r]esolved deltas, because r was replaced with the cursor all the time ("[]esolved deltas") and then with the "resolved deltas" again. Same for progress lines, every time the cursor was blinking there was a new line, that's what I am assuming.

• I had to clear most of the lines that were produced when curl was using the --progress-bar flag. If you want to reproduce this, see below.

• I fixed the file by manually editing the asciinema cast file with VIM, because all the tools available were too ridiculously buggy to use and/or too complicated and did not respect the given delta time (as "maximum time between lines"), apparently. My fixing script that you can use after you've removed all lines in the asciinema file is located here, with some example output: https://gist.github.com/cookiengineer/41d667b616abb0de0b8dda557ff4414f

Example to reproduce:

• First, throttle your internet connection. Use your smartphone via tethering, for example. Edge/2G/3G should suffice for demo purposes.

• Execute this bash script, it will download the big release file from github to /tmp/lycheejs-runtime.zip and will extract it to /tmp/lycheejs-runtime-extracted afterwards:

#!/bin/bash

RUNTIME_FILE="lycheejs-runtime.zip";

# XXX: Automated download of lycheejs-runtime.zip
if [ ! -f /tmp/lycheejs-runtime.zip ]; then

	echo " (L) > Installing lychee.js Runtimes ...";
	echo " (L)   (This might take a while)";

	DOWNLOAD_URL=$(curl -s https://api.github.com/repos/Artificial-Engineering/lycheejs-runtime/releases/latest | grep "browser_download_url" | grep "$RUNTIME_FILE" | head -n 1 | cut -d'"' -f4);
	DOWNLOAD_SUCCESS=0;

	if [ "$DOWNLOAD_URL" != "" ]; then

		DOWNLOAD_SUCCESS=1;

		cd /tmp;
		echo " (L)   curl --location --retry 8 --retry-connrefused --progress-bar $DOWNLOAD_URL > /tmp/lycheejs-runtime.zip";
		curl --location --retry 8 --retry-connrefused --progress-bar $DOWNLOAD_URL > /tmp/lycheejs-runtime.zip;

		if [ $? != 0 ]; then
			DOWNLOAD_SUCCESS=0;
		fi;

		if [ "$DOWNLOAD_SUCCESS" == "1" ]; then

			mkdir /tmp/lycheejs-runtime-extracted;
			echo " (L)   cd /tmp/lycheejs-runtime-extracted";
			echo " (L)   unzip -qq ../lycheejs-runtime.zip";
			cd /tmp/lycheejs-runtime-extracted;
			unzip -qq ../lycheejs-runtime.zip;

			if [ $? != 0 ]; then
				DOWNLOAD_SUCCESS=0;
			fi;

		fi;

		if [ "$DOWNLOAD_SUCCESS" == "1" ]; then

			# XXX: Do nothing, just a test.
			echo "Well, fuck. Didn't work.";

		else
			rm /tmp/lycheejs-runtime.zip 2> /dev/null;
		fi;


		if [ "$DOWNLOAD_SUCCESS" == "1" ]; then
			echo -e "\e[42m\e[97m (I) > SUCCESS \e[0m";
		else
			echo -e "\e[41m\e[97m (E) > FAILURE \e[0m";
			exit 1;
		fi;

	fi;

fi;

[cookiengineer]

@nbedos Lucky me, I found the ascii recording files and all steps in between the modifications on my old laptop. I added them to the gist: https://gist.github.com/cookiengineer/41d667b616abb0de0b8dda557ff4414f

File Descriptions (from the README in there):

01-install-lycheejs.original.cast contains the original
recorded asciinema file. It contains all unnecessary lines
with curl --progress-bar entries, reset characters and
multiple lines for identical content due to (probably) the
terminal cursor blinking all the time.

02-install-lycheejs.edited.cast contains the identical
content, but has all the reset characters relevant lines
manually removed (via editing with VIM and a lot of macros).

03-install-lycheejs.fixed.cast is the file outputted
by the asciinema-patcher.js file, so that the timeline
is fixed and entries with a delta beyond the given
idle_time_limit of the initial recording file are correctly
ignored and merged together, so that the final timeline
is correct.

With the current HEAD of termtosvg, I could reproduce the following:

# Result is 3.8MB big (curl progress bars, reset characters are in there)
termtosvg render 01-install-lycheejs.original.cast 01-install-lycheejs.original.svg;

# Result is 413.5kB, unnecessary lengthAdjust, text-decoration, textLength properties
termtosvg render 03-install-lycheejs.fixed.cast 03-install-lycheejs.fixed.svg;

# After manual editing, the file can be reduced to around 370kB (without! grouping)

I also dumped all SVGs in their steps and pretty printed variants for comparison in this separate gist here: https://gist.github.com/cookiengineer/97082877faec8bd95b1378a7b7620237

After step 3 there was an additional step 4 where I was modifying the SVG file manually, and removing all unnecessary properties of all <text> nodes that I could find. I didn't use any <g> here, so I don't know the potential benefit of grouping nodes together, meaning colors, bg colors, styles, etc could potentially be reflected by a couple of groups.

Pretty printing (for the sake of using git diff) was done using cat file.svg | xmllint --format - > file.pretty.svg.

[nbedos]

@cookiengineer Thanks for the detailed report. I've made a few improvements to termtosvg regarding file size. Nothing drastic but the size of the examples at https://github.com/nbedos/termtosvg/tree/develop/examples are reduced by 10% to 25%.

I've also added a command line option to enforce a minimum frame duration. Basically it merges consecutive frames together and helps control the size of recordings involving calls like curl --progress-bar that make lots and lots of updates to the terminal screen. See the man page for more details.

I've not made a release yet, but all these changes are on the "develop" branch.

[nbedos]

I've not made a release yet, but all these changes are on the "develop" branch.

Those changes are included in version 0.6.0 which was published a month ago, so I'm closing this issue.