encfs 1.8 slow compared to encfs 1.7.4

[Aikhjarto]

I just did an upgrade from encfs 1.7.4 to the newly released 1.8. I use encfs in reverse in combination with rsync to copy my backups to a second location.
My test folder has a size of 5 GB with 150000 files in it. With encfs 1.7.4 the copying the test folder was done in 15 sec. Runing encfs with rsync on the same files (again, no changes) with encfs 1.8 took about 5 min with high CPU utilization of encfs all the time.

Has anyone an idea, why the new version is that much slower?

[rfjakob]

EncFS 1.8 uses per-file unique IVs in the default reverse configuration, that may be the reason.
Please try to set it up in expert mode and disable unique IVs. If the performance hit is that big we should probably disable it by default.

[Aikhjarto]

Just did a
encfs --reverse $(pwd) ~/test
and selected the expert mode.

It says:
--reverse specified, not using unique/chained IV

So, IMHO unique IVs are not used in reverse mode.

[rfjakob]

Yes, the message is unclear. Actually, only "chained IV" is disabled!

[rfjakob]

By the way, there is an easy way to check: If every encrypted file is 8 bytes larger than the original, uniqueIV is definitely enabled.

[Aikhjarto]

Actually the encrypted files did not change with the update from 1.7.4 to 1.8 (i used to old .encfs6.xml file). Rsync did not report any changes.

Just did the expert mode and answered the question as follows:

Enable filename initialization vector chaining? 
[y]/n: n

Enable per-file initialization vectors?
[y]/n: n

External chained IV disabled, as both 'IV chaining'
and 'unique IV' features are required for this option.
Enable block authentication code headers
y/[n]: n

Add random bytes to each block header?
Select a number of bytes, from 0 (no random bytes) to 8: 0

Enable file-hole pass-through?
[y]/n: y

This actually produces the same setting in the xml file as the one I have from version 1.7.4. Only, of course, encodedKeyData, saltData, kdfIterations, and creator fields are different.

[rfjakob]

Thanks for all the details, so uniqueIV is not the reason. I will investigate this today in the evening.

[rfjakob]

Ok, I think I got it. Kernel caching for reverse mode was disabled in eaf3fc6 because it made a self-test fail. The absence of caching causes the slowdown.
I have re-enabled caching and modified the test to explicitely pass --nocache when mounting. Here, all tests pass and rsync is fast again.

Please

git clone https://github.com/rfjakob/encfs.git

and verify. The fix will then be included in 1.8.1.

[Aikhjarto]

That did the trick. Now 1.8 is as fast as 1.7.4 in reverse mode again.

Still i wonder, whats the reason why kernel caching must be disabled for the test you mention?
I'm worried if i might hit such a case while using encfs.

[rfjakob]

The kernel caches file metadata for 2 seconds. This means that the encrypted view will be up to 2 seconds behind and rsync may not pick up the changes. How do you trigger the backup job? Does it modify the data (dump a database or something like that)?

[Aikhjarto]

As this would only apply for rapidly changing files, this is no issue for my backup job for two reasons,
First, my backup is based on storeBackup which does not rely on metadata do be consistent. Second, prior running encfs, I do some checks with "du" and "df" if there is enough space left to prevent rsync from struggle. These checks most likely last longer then 2 seconds.

[rfjakob]

Yes, that looks fine.
However, I will investigate if there is a way to disable caching without the massive performance hit. The two second delay can actually cause trouble for some use cases. I will keep the bug open as a reminder.