15 November 2026
From a message to an event
A message arriving from the broker is not yet data you can store. It is bytes, on a topic, delivered at least once, possibly late, possibly twice. The ingestion brick turns that into an event.
What an event is
An event is a small record of something that happened:
{"device": "greenhouse-3", "metric": "temperature", "value": 21.4, "time": "2026-11-13T09:14:02Z"}
Two properties matter. It is a fact: it is never edited afterwards. A later reading is a new event, not an update. And it carries its own time, so it can be placed in history wherever it arrives.
Three problems ingestion must solve
Validation. A device with a bug sends "value": "twenty". Ingestion decides what to do with it: reject it, and
keep a count of rejects so the problem is visible.
Duplicates. With QoS 1 a message can be delivered twice. If every duplicate is stored, a sum or a count is wrong. The usual answer is to give each event an identity (device plus a sequence number or timestamp) and make the write idempotent: writing the same event twice leaves one row.
Two clocks. The event time is when the sensor measured. The ingest time is when your backend saw it. A device that lost its network for ten minutes sends old readings in a burst, so arrival order is not time order. Store the event time, keep the ingest time too, and never assume the stream is sorted.
Why this is its own brick
Putting parsing in the broker, or in the database layer, hides these decisions. As a separate small service, ingestion has one contract: bytes in, valid unique events out. That contract is easy to test, and easy to blame when a number looks wrong.