My boiler monitor kept going quiet, and the network was fine.
The whole point of the boiler monitor is that it tells someone when the fire goes out. That is its entire job: read a few temperatures, watch the pumps, and send an alert before the house gets cold. So the failure that actually got my attention was not a bad reading or a crash. It was the alerting going quiet. I found out the way you never want to find out: my brother texted me to say he had not gotten a Telegram alert.
A monitor you cannot trust to speak up is worse than no monitor at all, because you have quietly stopped watching yourself. The surprise, when I dug in, was that the device was still running and sending metrics, just not sending any alerts. This post is about chasing down a memory problem, and what it takes to keep a small device honest and online without a road trip out there.
The setup
A while back I wrote up why I built a wood boiler monitor for my brother Theo. If you want the backstory, the sensors, and the why, you should read that one.
This one is going to be way more technical, about the part I skipped: keeping the thing connected.
Some context for this story: it is a LILYGO T-SIM7080-S3 sitting out at an outdoor wood boiler, a drive from my place, so I do not get to walk over and poke it when something breaks. It has two ways off the property: WiFi, which is on Starlink, and LTE straight through the onboard SIM7080G modem, which should in theory be the backup if the power (and with it the WiFi) goes down. Data goes out as MQTT over TLS, with Telegram alerts on top via the API and a firmware update check every few hours.
Why WiFi is primary and LTE is only the backup
The WiFi at the boiler is bad. It sits around -88 dBm, which on paper is the kind of number where you expect constant drops. In practice it holds. Weak but steady beats strong but flaky, and once the ESP32 is associated the link is pretty stable.
LTE loses on a few things that matter out there:
- Data cost. The IoT SIM is about $15 (CAD) a year for 1 GB, which is actually plenty for readings every 60 seconds plus firmware updates. But WiFi is free and unmetered, so there is no reason to spend a metered budget while it is up.
- Latency. WiFi is instant once associated; cellular is not. A cold modem has to power up, attach to the network, bring up the link… That is seconds of work before the first byte moves, and on the weak signal even more.
- Signal. Out there, you cannot really take a phone call if you are not in range of the cell booster or using WiFi Calling. So our little monitor has a hard time finding service as well.
So the rule is simple: use WiFi whenever it is up, and only fail over to LTE when WiFi is genuinely gone. The failure I built the fallback for is not usually the WiFi itself. It is a power or Starlink outage taking out the whole house, and cellular is the one path that does not care because it does not go through the house. Whether that theory survived contact with reality is something I will come back to at the end.
The SIM7080G, and where the time actually goes
On paper the SIM7080G is lovely. Low power, Cat-M and NB-IoT, a built-in TCP and MQTT stack so the ESP32 does not have to run the TLS itself. In practice, the modem is a second computer bolted to the first one that you talk to over a serial port in a 1990s command language (the AT commands below), and that is where a disproportionate amount of debugging and testing time went.
Powering it up is not “set a pin high.” You cannot just enable the module and start talking. The modem wants a specific power-up cycle, and then it still comes up on its own schedule. My bring-up code just pulses and retries until the chip answers, bounded by a time budget so a dead modem fails cleanly instead of hanging the task long enough to trip the watchdog (and reset the main microcontroller):
while (!modem.testAT(1000)) {
esp_task_wdt_reset();
if (millis() - start > WAKE_BUDGET_MS) { // 60 s
Log::error(TAG, "Modem AT timeout");
return false;
}
if (++retry > 6) {
Log::info(TAG, "PWRKEY pulse");
digitalWrite(PIN_MODEM_PWR, LOW); delay(100);
digitalWrite(PIN_MODEM_PWR, HIGH); delay(1000);
digitalWrite(PIN_MODEM_PWR, LOW);
retry = 0;
}
}
The pulse timing is not arbitrary; the datasheet wants a pull of roughly that length, and shorter ones were flakier when I tried them.
The URC that silently vanishes. This one cost me almost a full weekend. The SIM7080 runs its own MQTT client, and inbound messages arrive as an unsolicited result code on the serial line:
+SMSUB: "owb/cmd","{\"reboot\":true}"
The library I use for the AT interface (TinyGSM) has a URC handler for the SIM7080, but it does not know about +SMSUB:. So the line arrives, the library does not recognize it, and it gets quietly dropped into the void. From the outside everything looks fine: you subscribe, the broker confirms, the broker delivers, and nothing ever reaches your callback function. No error. Just silence on the inbound path.
The current fix is to stop relying on the library for this and drain the serial port every loop, matching the +SMSUB: line and parsing it by hand:
// Format: +SMSUB: "<topic>","<payload>"
char* p = line + 8;
if (*p != '"') return;
p++;
char* topicStart = p;
char* topicEnd = strchr(p, '"');
if (!topicEnd) return;
*topicEnd = '\0';
// ... same again for the payload between the next pair of quotes ...
MQTTClient::dispatchInbound(topicStart, payloadStart, payloadLen);
Two gotchas hide in that innocent-looking parse. First, a URC does not wait for a quiet moment. One can land in the middle of another command’s response, so the same +SMSUB: matcher has to run inside the wait loop for every AT command too, or you drop inbound messages the moment the device is busy publishing.
Second, an empty payload (some commands from my app arrive with no body at all) has a degenerate "topic","" form that a naive parser throws away. Handling those zero-length payloads had to be explicit.
The worst failure is the one that looks healthy. There is a state the SIM7080G gets into where every status register you would think to check comes back green. The modem is registered, signal strength is fine, the MQTT session reports connected. And +SMSUB: URCs just stop arriving. Nothing in the status tells you the inbound path is dead, because as far as the modem is concerned it is not.
There is no command (or at least I did not find one so far) that fixes this behaviour. The only thing that reliably brings it back is a hard power-cycle of the modem itself. As of now, a watchdog watches the inbound path specifically: if the modem claims to be healthy but has gone silent past a specific threshold, it hard-resets the whole thing.
TLS is the expensive part
Everything leaves the boiler encrypted: MQTT over TLS, Telegram over HTTPS, OTA over HTTPS. On a laptop or smartphone that is free. On an ESP32 with something like 512 KB of RAM to share across the entire program, TLS is the single most expensive thing the firmware does. One WiFiClientSecure handshake wants roughly 30 KB of heap, and mbedtls (the TLS library underneath) needs a good chunk of that to be contiguous, not just free somewhere. The S3 does have 8 MB of slower PSRAM to offload bigger buffers to, but the handshake wants its working memory in internal RAM, so PSRAM does not buy a way out here.
That contiguous part is the whole trap. I wrote about heap fragmentation in the boiler post already - the parking-lot version where you have plenty of total free memory but not one block big enough for the next thing. TLS is where that bites hardest.
The mistake I made first was a fresh WiFiClientSecure for every connection. It only works for a little while. But every make-and-destroy cycle leaves the heap a little more fragmented, and after a handful of them there is no 30 KB block left for the next handshake even though there is plenty of free memory in total. The device does not crash cleanly; it just stops being able to connect.
The fix is boring and it is the right one: allocate the TLS client once, as a static, and reuse it for the life of the program. One for MQTT, one for Telegram:
WiFiClientSecure TelegramModule::_secClient; // one instance, reused forever
No per-connection, so no fragmentation from that path.
The catch with reusing it is that the client holds TLS state between connections, and some of that state is sticky in a way that messes things up again. If a handshake fails partly, WiFiClientSecure keeps the mbedtls context around with whatever cert pointers it had. Call setCACert or setCertificate again on that still-allocated context and it is silently ignored: the next handshake reuses the stale config. On the broker side you get
peer did not return a certificate
which is really confusing the first time, because you are very much setting a certificate. The fix is to stop() the client before setting the cert again:
_wifiClient.stop(); // release the mbedtls context first
_wifiClient.setCertificate(_clientCert);
_wifiClient.setPrivateKey(_clientKey);
And when fragmentation wins anyway, because after weeks of uptime it just does, there is one last resort. If the allocation has failed 3 times in a row and the largest free block has dropped under the threshold, the firmware stops fighting and reboots itself:
if (_retryCount >= 3 && ESP.getMaxAllocHeap() < 40000) {
Log::error(TAG, "TLS alloc failed 3x with low heap - rebooting to defrag");
ESP.restart();
}
A deliberate reboot out there costs a few seconds nobody really notices. A broken device that needs a drive to power-cycle costs an afternoon. Rebooting on purpose is the cheap and fail-safe option, and that idea comes back in the next part.
So what was actually going wrong
Here is the part that took me a while to accept: the connection was fine.
WiFi associated with its usual weak-but-steady signal, the MQTT client seemed to be connected. And still nothing going out. Same problem the modem had a couple of sections up, only one level higher: the whole device was confidently offline. It believed it was online, and it had run out of memory to actually send anything.
That is what memory fragmentation does over long uptime. Not a clean crash, not a disconnect you can catch. Just a slow slide until the next thing that needs a contiguous block - a publish buffer, a TLS handshake for reconnect - cannot get one. The client still says connected, because the socket is still open. The publish just fails quietly, because there is no room to build the message. From the outside, it is silence.
You cannot fix that by trusting anything the device says about itself, because everything it says is “fine.” So the firmware stops trusting it. If the MQTT client claims to be connected but nothing has successfully gone out past a specific threshold (as of now 10 minutes), a watchdog tears the session down and reconnects from scratch:
if (_lastSuccessMs > 0 && (now - _lastSuccessMs > WATCHDOG_MS)) { // 10 min
Log::warn(TAG, "Watchdog: no activity - forcing reconnect");
_client.disconnect();
_wifiClient.stop();
}
And behind it sits the heap floor from the last section. When that forced reconnect tries to handshake and there is no contiguous block left for it, the low-heap reboot fires, the device comes back with a clean heap, and it reconnects in a few seconds. The reboot is not the failure; it is the recovery.
It is not elegant, but it is honest about the constraint it lives under, and the alerts start flowing again on their own.
I did also harden the actual network path while I was in here, because a remote device deserves belt and suspenders. There is TCP keepalive on the socket (setsockopt with a 30 second idle, 10 second probes, 3 strikes) so a genuinely half-dead link gets noticed in tens of seconds instead of never. And a hard 10 second cap on every blocking HTTPS call so a slow Telegram send cannot sit in the main loop long enough to starve the MQTT keepalive. Those were real improvements. But I want to be straight about it: they were not what had gone wrong. What had gone wrong was memory fragmentation.
Has the fallback ever actually fired?
Time to make good on the promise from the top. I put a lot of hours into the cellular path; the modem section up there is the short version of a long fight. So here is the fair question: out at the real boiler, has any of it ever actually saved the day?
As far as I can tell, no. Not once.
That is less depressing than it sounds. The fallback only does anything when WiFi is gone, and the ugly-but-stable WiFi out there almost never goes fully away. The few times it might have, I was not watching, and a quiet successful failover does not send up an alert to say it earned its keep last night. A backup path that works perfectly is invisible.
So how do you test something that almost never fires on its own? You force it. The cleanest trick I found is to point the WiFi config temporarily at an SSID that does not exist (newer firmware versions have a wifi.enabled switch for it in the config). The device tries, fails, gives up on WiFi, and walks through its failover: standby, activating, modem up, and then it starts pushing metrics over LTE. I have done that on the bench and once on site, standing next to it, and both times the mechanism did what it should.
But I want to be honest about what that does and does not prove.
- I forced it. A real outage, or a half-associated access point, might not look as clean as flipping to a fake SSID.
- The time I tested on site, nothing happened to be alerting. So the exact path I should care about most was only tested on the bench.
- Cellular coverage out there is bad. The backup link has weak signal at exactly the spot it would have to work, and more testing and maybe some antenna optimization are still pending because of that.
Since I cannot count on ever seeing it fire, I built it to fail safe instead of to look good. Activation runs on a timer: if the modem cannot get online within about thirty minutes, it hard-resets and drops back to standby rather than sitting there silent. There is also a hysteresis on the transitions so it does not flap between WiFi and LTE. The rule I settled on is that a fallback I cannot watch has to have a worst case of “quietly gives up and retries,” never “bricks the one device I would have to drive out to reach.”
So, in theory it works, and I have bench-proved the pieces work. I have not watched it carry a real alert through a real outage. It is insurance I have partly, not fully, and some more tests and optimizations are coming.
Where this goes next
The to-do list out of all this is short. Do real failover testing at the site, which mostly means dealing with the bad cellular coverage before the fallback is worth trusting. And keep watching the heap over long uptime, because “reboot before you go quiet” is a safety net, not a fix, and I would rather it fragmented slower in the first place. I already send the free heap and maximum allocatable block out via MQTT and it seems to have stabilized. The longest uptime so far was around 14 days, until I pushed a firmware update and rebooted the device. And the memory metrics looked healthy by then.
None of it is solved. It runs, it has not missed a real alert since the fix, and I keep finding new ways it can quietly let me down.
I also had the idea of integrating LoRaWAN and putting a central hub with better WiFi and cellular uplink somewhere else. But that is just theory for now and would be another device and failure domain to take care of.
The firmware is all open if you want to see how any of this actually looks in code: thesada-fw on GitHub.