For the past few years I have been operating a small number of Arduino Ethernets with DHT22 sensors, push data into Graphite. They have worked reasonably well, but had one major issue. Graphite required that the data flow include the Unix Epoch time and the Arduino time counter over flows every ~50 days.
The graphite data needs to look like this:
topic value epochtime
or:
house.temp.crawlspace 17.600000 1590181123
So with one Arduino, I would just occasionally unplug the device and
then on startup it would sync with the local NTP server and use
millis() to figure out how long it had been since that initial sync.
Sure, I had ideas of ways to work around this, like setting up the
watchdog to reset the Arduino automatically, but that still has
problems.
My goal was to eleminate the need for the Arduino to know what time it
was at all.
Initially I was going to write a small proxy to handle adding the time
into the data stream.
This would give me a small project to practice my C skills and learn
about network sockets.
As this was percolating in the back of my mind I learned about MQTT at
a talk by Jan-Piet Mens
gave at BSDCan 2019.
As I started researching more things I could do with MQTT, I also learned about this fantastic HomeBridge Plugin called MQTT-Thing. This convinced me it was time to setup an MQTT broker and start connecting various things to it.
For the past few months I have been using JP’s
mqtt2graphite to take the
data from MQTT and shovel it into Graphite.
Earlier this month I performed and upgrade and I noticed that data
stopped flowing.
After trying a frustrating day beating my head trying to figure out
why data was not coming out of mqtt2graphite, I noticed with tcpdump
that it was not even putting any data on the wire.
Which eventually lead me to add debugging around the sendto() call.
This lead to me rediscovering a big flaw in the Paho MQTT library for
Python, and the worst part is, it is a flaw I have run into before!
Low and behold it is a small world, I find a issue raised by JP against
the Paho MQTT Python git repo about exceptions not being raised in
callbacks
#365.
So I used try/except to see what was going on and modified the code to
exit when an error with the socket happens. Somewhere along the Python
3.x train sendto() was changed to require bytes and will not accept a
string. So I have made the required
changes
and would send a pull request over to JP, but I noticed he has archived
the repo for mqtt2graphite, but I hope others will find my changes to
be useful.
The repo is here: https://github.com/brd/mqtt2graphite
So thanks JP for the work you have put into mqtt2graphite and for teaching me about MQTT in the first place!