Resume after power failure?

[cosmos72]

Hello, I am the author of fstransform https://github.com/cosmos72/fstransform a tool to change the file system type on a device, in place and without need for a backup. Your program looks very interesting, and had to solve some of yhe same challenges I met in mine.

If I understand correctly the documentation, luksipc cannot resume a conversion after a power failure, because it did not have time to create the"resume.bin" file contaning the pointers and the shadow block.

If you are interested, I have an idea to improve you program, so that it can resume a conversion even after a power failure.

In summary, the idea is:
Write the first block and the pointers into resume.bin before writing the LUKS header,
then encrypt the device backwards, i.e. first reading the (N-1)th block and encrypting it into the N-th block, then reading the (N-2)th block and encrypting it into the (N-1)th block, and so on...
Every time you encrypt one block, update the pointers in resume.bin and fsync() everything.

I did something similar in fstransform (actually a bit more complicated, I did not have the luxury of processing blocks sequentially) and it CAN resume a conversion even after a power failure :)

[johndoe31415]

Hi there,

you are correct about the observation that currently, power failure during a luksipc conversion means that you lose the point where you can continue. Someone even made a patch once that wrote the resume file after each and every block and synced disks. Obviouly, as you also correctly note, for front-to-back conversion this means that this is a rather large performance hit (since the resume file needs to contain the contents of a whole block).

Back-to-front conversion avoids the problem of a large resume file if the converted block size is less than the size of the LUKS header itself. Since the LUKS header is about 2 MiB, this seems like it would be sufficiently large to avoid frequent calls into the kernel. While currently that buffer is 10 MiB, I think the additionally introduced context switches will not introduce a big performance hit.

I've had that on my radar for a while, but not tackled this so far. The reason is this: I do rigorous testing of luksipc to behave well even in the face of unexepected errors. This is a process that is tedious and painful. Rewriting the conversion to go back-to-front instead of front-to-back means rewriting the whole core of the application which means that I would need to be extremely vigilant in the implementation in order to fix newly introduced regressions. This also means that this isn't something that I want to implement as a side project, but with full focus -- and currently I just didn't find the time to do this yet.

Another issue is the resume files and their compatibility. To keep things simple, I would drop compatibility between v1 and v2 resume files completely. Hm, thinking about this, I don't think people upgrade luksipc between disk conversions. Hopefully they don't. In any case, the resume file header already has a version field which allows me to detect older version.

About the ability to resume from hard disk failures, I'm not so sure to be honest: Let's say we use a copy block the size of the LUKS header to keep the number of kernel calls small. Let's furthermore assume that the writing of one such block has just been started (first sector). Then the power fails. Integrity of the whole 2 MiB block will be guaranteed if and only if that block is written to completion and the resume file is successfully synced to disk afterwards. I would say if the remaining power in the SMPS of the PC if sufficient to keep the computer running to write that block depends heavily on the actually used hardware. A solution would be to write and sync a resume file saying that it's about to start writing a block and then again one that says it's finished writing that block. In any case, I think the block that is just about to written is still, without manual interaction or heuristic detection of which part of the block has already been encrypted and which has not, questionable. I would love to see actual testing on this to see how a hard disk behaves in practice when the PC is shut down hard.

In any case, implementation of this is something that I have wanted to implement but just didn't get around to doing so far. I'll leave this bug open for now for the future. Thank you for your interest in luksipc and thanks for your suggestions. Have a nice day, Cheers, Joe