Reducing write load to SDcard on RasPi

I had two microSD cards ending up in trash in my Webcam Raspberry Pi in the past few months. If you think about it, the reason is quite simple: SDcards can only serve so many write cycles, for example about 10.000 cycles per cell with single level flash memory. When saving 1,5MByte of a webcam snapshot each minute, first as a temp file, then working on it to add a timestamp, then finally writing it, this leads to abot 5MBytes written - every single minute.

Even with good wear leveling algorithms you will write 7.2GByte each single day. It should be doubled because you need to erase-before-write on flash memory. On a 8 GByte SDcard, with much luck would lead to 5.000 days usage time. But reality looks different. The cheap memory cards use multilevel flash which can serve much less write cycles, about 1000. Even if the wear leveling works good, this would lead to data corruption in less than 1 1/2 years.

When the sysstat package is installed, you can see the throughput of your SDcard:

#iostat
Linux 4.4.26+ (CamPi)   20.11.2016      _armv6l_        (1 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10,81    0,00    4,28    0,29    0,00   84,62
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           3,08        54,06        24,00   78615488   34898399

All this in this time:

#uptime
 07:57:38 up 17 days, 13:01,  1 user,  load average: 0,78, 1,58, 1,05

Which means nearly 34 GByte written in 17 days, which isn't as bad as initially thought; "just" 2 GBytes per day. It was way worse before, as I already have disabled the Swap File which Raspbian enables by default. On a RasPi A+ with only 128MByte of RAM this swapfile gets used intensively as it simulates RAM. For some updates, it is necessary to re-enable it as otherwise the package can't be decompressed. But this is done very easily and can be disabled immediately afterwards again.

To disable the swap service, run these commands:

#sudo swapoff -a
#sudo service dphys-swapfile stop
#sudo systemctl disable dphys-swapfile

To turn it on temporarily, then use these commands:

#sudo systemctl start dphys-swapfile
#sudo swapon -a

Don't forget to disable swapping after the task again. Another good idea is to set the memory splitting differently. On RasPi A+ with RasPi Camera 128/128MB are recommended, but it works also with setting GPU memory to 64 MByte via raspi-config or /boot/config.txt. This lessens the write-load on the SDcard significantly as well.

My webcam picture every minute - first as temp.jpg, then copied over as final .jpg - leads to severe amounts of data written. So I used a simple fix: Create a RAM disk with 10 MByte, mount it as tmp within the www directory, and use that for storing the image:

#mkdir /var/www/tmp
#sudo nano /etc/fstab
-> add the line:
  tmpfs /var/www/tmp tmpfs nodev,nosuid,size=10M 0 0
#sudo mount -a
#df -h
# ln -s /var/www/tmp/snapshot.jpg /var/www/snapshot.jpg

The df-command should show the /var/www/tmp directory as tmpfs. Now I adapted the webcam.sh script to use this temp directory instead the www root; the index.html also refers to the new directory. Another fix is to give imagemagick the tmpfs directory as working directory by adding to the call:

-define registry:temporary-path=/var/www/tmp \

This works flawlessly without problems.

Now the stats look a bit better:

 $ iostat
Linux 4.4.32+ (CamPi)   20.11.2016      _armv6l_        (1 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11,52    0,00    4,81    0,19    0,00   83,48
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           2,61        47,72         5,46      77261       8844
 $ uptime
 08:34:27 up 28 min,  1 user,  load average: 0,10, 0,19, 0,18

I'm still looking where I can save some disk write accesses, as there still gets a lot of data written. But I'm down by a factor of 10 now, from 14.4GB per day (erase+write) to less than 1.4GB per day. This will lead to much longer lifetime of the SDcard.

To find out which processes write to the SDcard, you can use iotop:

#sudo apt-get install iotop
#sudo iotop

The switch --accumulate can be helping as well.

DIY ATmega328 Clock - improved power saving

I built a clock as fun and training project two years ago. There is always room for improvement - first I had a 7 segment LED which I multiplexed at 1kHz and used the time in between to sleep the µC. I tested several RTCs, from DS1302 over 1307 to the DS3231. The DS3231 is in place with a Nokia5110 display now for half a year and led to significant better time keeping and less power usage.

ATmega328 based DIY Clock.

Several further sensors are attached - the Si7021 measures temperature and humidity. Cool thing about the IC is that it does its measurement and goes to sleep immediatly, consuming less than 1µA.
Another sensor is a BH1750 which is used for only activating the display when there is enough light to read it. As soon as it is dark, the display gets disabled, further dropping the current drawn.

Today I added two further improvements to bring the power consumption down a little more. For one, I modified the Wire-library of the Arduino GUI to not activate the I2C pullup resistors. On those sensors, there are Pullups in place already.
Find your arduino-directory, go to hardware\arduino\avr\libraries\Wire\src\utility and open twi.c - there you can search for "pullup" and comment out both lines where the SDA/SCL pins are set to 1 (which enables the pullup resistor on an input pin).

Power consumption via 1R shunt on the oscilloscope.

This led to the current drop to 0µA every now and then.
So I decided that 100kHz I2C speed may be a little slow and keep the µC awake longer than necessary. So I added the line
TWBR=12
at the end of the setup()-routine.

This value fits for 16MHz µCs and results in 400kHz I2C speed. Looking at he graph it can be seen that the I2C reading is now taking only a short fragment of a second instead of nearly half of it as it did before. The values are close to 1mV for 1mA, but with such low values, this setup is distorted by noise a lot and doesn't show the precise load values. The multimeter shows peaks of ~3mA and sleep periods with real 0µA.
Before these changes, with a proper low-quiescient-current ultra-low-dropout regulator like the MCP1700, the LiIon battery with 2400mAh lasted for exactly three months. Let's see how long the new battery will last this time! I'll update the article when the battery is empty.

Power supply for µCs via LDOs

There are plenty of LDOs on the market - some cheap, some expensive, some with only three terminals, other with 5 or more. So how to decide upon a good LDO (Low DropOut regulator)? And why use a LDO when it literally burns energy within it's pass transistor? A switching regulator feels way more efficient as it only takes as much energy from the source as is needed on the output?

This all depends on your usecase. If you have no constraints in your power supply scheme like when using a USB power supply or a 12V switching power supply, you may use a switching regulator. It has a little ripple on the output voltage which may be disturbing in some cases, for example if you want to use an ADC. But it won't get too hot, and 12V down to 5V or 3.3V can be done quite efficient for medium loads.

Here's the point: It depends on your load. A switching regulator won't adapt as fast to changing loads, for example when a µC goes to deep sleep and suddenly wakes up. There the load will jump from a few µA p to 100mA or more. For the time when the µC is sleeping, the quiescient current of the switching regulator is usually many times higher than that of the µC. If the µC is sleeping long time and only wakes up every so often - or your wake-up load is below for example 10mA-, a LDO is the more efficient choice.

For my DIY clock and my mobile ESP8285-based sensor I wanted the most efficient LDO possible. The newest and most promising ICs are in form factors that are nearly impossible to solder by hand. So it had to be SOT23 size ICs. (N.b., old ICs like *7833 or *1117 have a high quiescient current of ~5mA, usually making then unusable with battery driven gear.)

Most efficient LDOs for DIY use.

For the 3.3V with a load of 250mAh and even more I found several ICs:
- HT73xx (7µA Iq)
- HT78xx (8µA Iq)
- XC6206 (2µA Iq)
- MCP1700 (2µA Iq)


I built up small PCBs which I could solder in front of my Clock and ESP.

They all delivered the load and voltage as expected. Differences became obvious, though.
The XC6206 is the cheapest of all those ICs. Unfortunately, it has a dropout voltage of max. 680mV at 200mA load. While this is great when your power comes from a USB supply, it means that a LiIon battery can be used only down to 4.0V! Also, the XC6206 becomes unstable very soon when driven by a LiIon battery with a voltage range of 3.0-4.2V. It is very sensible to load changes as well and starts to swing easily. I needed to provide a polarity protection as even short reverse polarization leads to a burnt chip.

Power consumption with different LDOs.

While the HT78xx (500mA max) and HT73xx (250mA max) also seemed to give stable performance, they didn't cope well with the load profile of sleeping 290s at <25 µA and then wake up for ~600ms at 70mA with short spikes of 250mA and more. Same behaviour shows with the clock which constantly switches between ~2mA and ~100µA. The datasheet has no graphs or values for the supply current with loads bigger than 40mA. The reason seems to be that the load on the battery significantly increases. The battery lost more than 15mV per day with these ICs. They're great when there is a wall plug involved or a huge battery and other big power hungry components, but for the ultra low current consumption I need they just don't cut it.

So the MCP1700 is the most stable and real economical LDO in this test field. Battery voltage drops about 5mV per day, leading to a runtime of roundabout half a year with my 900mAh LiIon battery.

My next tests are with lowering the voltage to 3.0V. The HT7330 showed the same behaviour as the HT78xx/HT73XX with 3.3V before and put a too big load on the battery. Interesting effect - after a runtime of more than one day, the HT78xx became much more stable and reduced the power consumption to values close to the MCP1700 ICs. I'm still waiting for MCP1700-302, I hope to shave off even more µAs to prolong the runtime with them.

Further improving powersaving with ESP8266 and LUA / NodeMCU

I already optimized my NodeMCU firmware - LUA code to send data and then go to sleep after less than 10 seconds. That now takes a bit more than 1.2 seconds. That was a great leap forward to run a mbile sensor on a single 14500 LiIon cell with 900mAh for over a month.

The first improvement was done by moving the DeepSleep call from the Close Connection handler within the "http-code" to the On Receive handler. This led to deep sleep at less than 50µA after about 1.2-1.6 seconds.

Going to sleep after reiception of the data has been
acknowledged by the server.

Today I tried to directly go to sleep after calling the send-command with the buffer of my data. This led to no data being sent at all - the sending is done asynchronously and the DeepSleep command just interrupts it before even being started.

The solution is a timer call. Maybe it's possible to shave off a few more ms, but I am sataisfied with the result so far.

Directly after the send-command add:
tmr.alarm(0,150,1,function() node.dsleep(deepsleep_time*1000) end)

So the ESP is going to sleep after 150ms after the send command. It works and successfully delivers the data to the server, so now the wakeup time is down to roundabout 0.6s. This means nearly doubling the runtime on a battery cell! (Another ~100ms are saved by using a ESP-01 with black PCB instead of ESP-12E, by the way.)

When calling deepsleep with a short timer after the send
command, the wakeup time becomes significantly shorter.

Update 24.09.2016: With trial-and-error I could go down to 120ms instead 150ms for the DeepSleep-timer-call. While this doesn't seem much, it is saving 5% of power. instead of 600ms, we now have 570ms of awake-time, summing up to 45,6mA used instead of 48mA. During the real 290s DeepSleep, the power used sums up to 7.25mA. Average power consumption is down to 181,88µA instead of 190,12µA.
Also, the LDO is now a MCP1700 with 3.0V output, which should give another reduction. It helps using the full voltage range from 4.2V down to 3.0V of the LiIon battery.

Different response times with different ESP modules

While optimizing the ESP-based temperature and humidity sensors for my home I stumbled upon a weird issue: The different modules show different times to connect to Wifi and send the data. This was a real issue with tha battery driven sensor as there the data was always sent after roundabout 4 seconds. Looking with a 1R and the scope at it, it became clear that the connection/wake time is much longer, even more than 10 seconds.

ESP-12F on an adapter PCB.

A bit could be solved in software. Going to deepsleep after reiception has been confirmed (on:receive-handler) brought the wake-time down close to the self-timed values of the board. Using a static IP borught the self-measured time down to 0.3-0.4s, being really awake for about 1.2s. The timing code used on my other modules revealed something:

ESP-01 old version (512kByte flash, blue PCB): 9-61s
ESP-01 new version (1 MByte flash, black PCB): 3,1s
ESP-12E: 2.95-3.05s
ESP-12F: 4.1s

It doesn't matter at which distance the sensors are to the base station. The connection time is quite stable around those values, only the old ESP-01 boards really vary between those values, most of the times at 15s though. The modules use the same firmware and the same code (one variable differs between the sensors which is their name which gets sent).

It is clear that the first versions may have a different µC stepping and/or worse layout than the newer modules. But I would have expected the ESP-12F to outperform all other boards, which clearly isn't the case. Maybe someone knows the reason for this behaviour. Leave a comment if you do!