Reducing write load to SDcard on RasPi

I had two microSD cards ending up in trash in my Webcam Raspberry Pi in the past few months. If you think about it, the reason is quite simple: SDcards can only serve so many write cycles, for example about 10.000 cycles per cell with single level flash memory. When saving 1,5MByte of a webcam snapshot each minute, first as a temp file, then working on it to add a timestamp, then finally writing it, this leads to abot 5MBytes written - every single minute.

Even with good wear leveling algorithms you will write 7.2GByte each single day. It should be doubled because you need to erase-before-write on flash memory. On a 8 GByte SDcard, with much luck would lead to 5.000 days usage time. But reality looks different. The cheap memory cards use multilevel flash which can serve much less write cycles, about 1000. Even if the wear leveling works good, this would lead to data corruption in less than 1 1/2 years.

When the sysstat package is installed, you can see the throughput of your SDcard:

#iostat
Linux 4.4.26+ (CamPi)   20.11.2016      _armv6l_        (1 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10,81    0,00    4,28    0,29    0,00   84,62
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           3,08        54,06        24,00   78615488   34898399

All this in this time:

#uptime
 07:57:38 up 17 days, 13:01,  1 user,  load average: 0,78, 1,58, 1,05

Which means nearly 34 GByte written in 17 days, which isn't as bad as initially thought; "just" 2 GBytes per day. It was way worse before, as I already have disabled the Swap File which Raspbian enables by default. On a RasPi A+ with only 128MByte of RAM this swapfile gets used intensively as it simulates RAM. For some updates, it is necessary to re-enable it as otherwise the package can't be decompressed. But this is done very easily and can be disabled immediately afterwards again.

To disable the swap service, run these commands:

#sudo swapoff -a
#sudo service dphys-swapfile stop
#sudo systemctl disable dphys-swapfile

To turn it on temporarily, then use these commands:

#sudo systemctl start dphys-swapfile
#sudo swapon -a

Don't forget to disable swapping after the task again. Another good idea is to set the memory splitting differently. On RasPi A+ with RasPi Camera 128/128MB are recommended, but it works also with setting GPU memory to 64 MByte via raspi-config or /boot/config.txt. This lessens the write-load on the SDcard significantly as well.

My webcam picture every minute - first as temp.jpg, then copied over as final .jpg - leads to severe amounts of data written. So I used a simple fix: Create a RAM disk with 10 MByte, mount it as tmp within the www directory, and use that for storing the image:

#mkdir /var/www/tmp
#sudo nano /etc/fstab
-> add the line:
  tmpfs /var/www/tmp tmpfs nodev,nosuid,size=10M 0 0
#sudo mount -a
#df -h
# ln -s /var/www/tmp/snapshot.jpg /var/www/snapshot.jpg

The df-command should show the /var/www/tmp directory as tmpfs. Now I adapted the webcam.sh script to use this temp directory instead the www root; the index.html also refers to the new directory. Another fix is to give imagemagick the tmpfs directory as working directory by adding to the call:

-define registry:temporary-path=/var/www/tmp \

This works flawlessly without problems.

Now the stats look a bit better:

 $ iostat
Linux 4.4.32+ (CamPi)   20.11.2016      _armv6l_        (1 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11,52    0,00    4,81    0,19    0,00   83,48
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           2,61        47,72         5,46      77261       8844
 $ uptime
 08:34:27 up 28 min,  1 user,  load average: 0,10, 0,19, 0,18

I'm still looking where I can save some disk write accesses, as there still gets a lot of data written. But I'm down by a factor of 10 now, from 14.4GB per day (erase+write) to less than 1.4GB per day. This will lead to much longer lifetime of the SDcard.

To find out which processes write to the SDcard, you can use iotop:

#sudo apt-get install iotop
#sudo iotop

The switch --accumulate can be helping as well.