Running zerofree after PiShrink can save additional space in compressed image

[xiegeo]

Suggestion: add zerofree as an additional step at the end of PiShrink.

I used a image file (3.27GB after PiShrink, 1.19GB zipped). After running sudo zerofree "$loopback" the image become 0.99GB zipped. Running zerofree was very fast on a shrinked image. It produced a sizeable saving for how image files are typically stored and transferred.

[framps]

That's a nice tip how to reduce the size of the image even more if it's zipped afterwords. Given the fact that the Linux philosophy is to create efficient tools for just one task and then use pipes to combine the tools I think pishrink should just be enabled to return the shrinked image to stdout.

That way following pipe can be used to achieve your task the Linux way 😃

pishrink <IMG> -- | zerofree | gzip > <zippedImage>

[xiegeo]

Thank you for your input.

It's not so simple since zerofree does not understant pi's partions. I only got it to work by copying how the script uses losetup. Also everything that modifies the file system at such a low level seems to work in-place (probably lots of out of order seeks and writes), so it's going to be hard to pipe.

[Drewsif]

I like the idea, it seems like it would be pretty easy to implement. I would probably lean towards making it make a large file full of zeros as to not make zerofree a requirement. But I will keep the idea in mind!

[jacobq]

It's not so simple since zerofree does not understant pi's partions.

It doesn't? I've had no trouble inserting an imaged microSD card and running zerofree /dev/sdX2.

I only got it to work by copying how the script uses losetup....

Isn't it as simple as the following?

LOOP_DEV=$(losetup -f)
losetup $LOOP_DEV -P whole_disk.img
zerofree ${LOOP_DEV}p2
losetup -d $LOOP_DEV

[xiegeo]

@jacobq It was hard for me because I never used losetup before. But if you can put that in a pipe, that will answer framps' question.

[rudiratlos]

here is well documented article, describing use of zerofree. There is also a referral to pishrink. Pls. consider to implement zerofree into pishrink.
https://unix.stackexchange.com/questions/565441/can-the-zerofree-process-be-performed-on-sd-card-image-file

tested this and it's working:
https://unix.stackexchange.com/questions/565441/can-the-zerofree-process-be-performed-on-sd-card-image-file

[framps]

@rudiratlos Looks like your link is dead 😢

[elmot]

I zerofreed my sd card before running the script. But that takes a lot of time and uses the card resource.
Gives 10%-15% of .xz file size decrease in my case.

One more similar idea - wipe swap file content. Or remove it completely and recreate it after the boot.

[dilyanpalauzov]

A faster alternative is to call e2fsck -E discard $loopback: #187

[jfikar]

The command e2fsck -E discard $device does not seem to discard the free space. Even e2fsck -f -y -D -E discard $device does not work.

On contrary, the zerofree or dd if=/dev/zero of=zerofile;sync;sync;rm zerofile works just fine. One can also zero the first FAT partition, to decrease the image size. Then dd has to be used.

One can probably also use fstrim on the mounted image, but I'm not sure if it does not depend on the trimability of the system storage.

There is also a possibility to create a sparse file at the end, if you are not compressing it by fallocate -d image

[jfikar]

This is indeed nice. But it seems to zero the free space only on the second partition with ext4, right? What about the first partition with vfat? The free space could be zeroed here as well for additional savings. Especially if the first partition is not shrunken at the end, while second partition is.

[Drewsif]

For sure. Step 1 is to zero free the last partition since we already mount it, then we can look at doing the other ones.

[ManiMatter]

hi all,

I am following a tutorial published here on how to create a backup of my SD card, which uses piShrink
https://www.tomshardware.com/how-to/back-up-raspberry-pi-as-disk-image

Is my understanding of above thread that zeroing the free space is not implemented as a standard in piShrink (yet), and you suggest to run zerofree on the image after piShrink has created it?

If its not a standard, any pointers as to how to run zerofree on the image would be greatly appreciated (will of course google myself to try to find the answer and read the manual).

[jfikar]

No, you should zero the file systems on the RPi SD card before using piShrink to achieve smaller image. I think the zerofree of the ext4 / second partition is already implemented. And zerofree does not work on the vfat /boot first partition anyway. So your steps are simply:

sudo dd if=/dev/zero bs=1M of=/boot/zerofile
sync
sudo rm /boot/zerofile

The dd ends with an error - out of free space, that's fine.

[ManiMatter]

Thank you, jfikar.

Makes sense that only the ext4 partition can be zeroed, since the other partition is read-only (vfat).

My understanding from your code is that it first creates a file called "zerofile" in the /boot/ folder fills it with 0s till disk space is used up, with junks of 1MB at the time written.
Then the cached data is forced to be written (sync), and finally we're again giving free the space by removing the zerofile.

Zerofree suggests that it's better than dd, because dd "
it is slow;
it makes the disk image (temporarily) grow to its maximal extent;
it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.
"
Quote from: https://www.kali.org/tools/zerofree/

My understanding is that the dd-approach that you suggested I can run from my normal linux.
For zerofree, however, I need to umount the disk and run it from recovery mode.

Therefore, my take is that zerofree may be faster, but the recovery-mode is a bit cumbersome.
Is that a fair summary?

I am looking at a SD card of 32GB, whereof probably 20GB are empty.
Any ideas how long it'd take with the suggested dd vs. zerofree?

Here is how i'd proceed if I went with zerofree rather than DD.
https://ubuntuforums.org/archive/index.php/t-2399268.html

Many thanks for your insights

[jfikar]

The /boot in Raspberry Pi OS is not read-only. It is RW.

$ mount | grep boot
/dev/sda1 on /boot type vfat (rw,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

The utility zerofree works only on ext2, ext3 and ext4. Not on vfat. I don't think it is faster than dd. Anyway, for /boot you don't have other choice.

I don't know, on what partition you want to use the zerofree. On /boot you can't, it is not supported. On / you don't need to, as PiShrink will run it for you.

[ManiMatter]

Many thanks jfikar, I learnt a lot from your explanations.

I'm running ubuntu 22.04 on my raspberry pi, but I think what you explained with the two partitions is true nevertheless.
In this case, I'll simply run the dd command as you had suggested on the "/ " partition.

I'd use this command, so I also see the progress:
$ sudo apt-get install pv
$ dd if=/dev/zero | pv | of=zero.file; sync; rm zero.file

And what I hope to gain from this is that the deleted files are wiped with zero's, so that when pishrink afterwards compresses it the deleted files are not considered.

Or are you saying that I don't even need to run the above command, since with pishrink having a go at it, this is already covered?

Many thanks for your patience :)

[jfikar]

I thought PiShrink is using zerofree now, but it is not. So your way should work. Sometimes pv and pipes slow down the dd (especially if you write only to RAM) and you can easily replace it by status=progress.

[ManiMatter]

So what I ended up doing is not to take an image in the end, because the image was >6GB even after shrinking (which is expected).

Instead I have connected dropbox to my ubuntu using rclone, and then I am using Kopia to backup key items from my OS to my drobpox.

I added a cron job that would write out all installed packages into the /home folder using apt-clone (every night at 2am)
I am backuping the /home, /etc, and the /var/spool/cron/crontabs (every night at 3am)

This way, I feel when needing to re-install my ubuntu from scratch, being able to re-built where I was before should be relatively straight forward.

Hope this helps others, too. Any suggestions / comments welcome.