Does the Data Source service Queue/Buffer MQTT Messages?

Introduction

Eratos supports self-service data sources for MQTT and AMQP protocols configured using the Data Source API. When data is sent to senaps.eratos.com message broker using these protocols, it is queued before being consumed by upstream services.

Buffering of messages is performed for several reasons:

Fault tolerance: Upstream services can be maintained without data loss. Further, buffered messages are backed by persistent queues, enabling the message broker to recover from unexpected downtime.
Enable quality of service: Data Sources provide fair ingestion rates to all publishers. Buffering messages before processing them ensures a 'messages per time instant' quality of service.
Separation of concerns: Upstream services parse messages as delivered, not worrying about the wire protocol or any other transport concerns.

Message Buffering Thresholds and Behavior

When publishing messages to the message broker, the following restrictions and thresholds apply on a per-data-source basis.

Buffering Pipeline Stage	Threshold	Details
Broker Receipt	Incoming Queue Depth: 50MB	Messages in the incoming queue are consumed by ingestion pipelines at different rates depending on platform load. You can inspect near real-time metrics or historical data.
Ingestion Processing	In-flight processing message threshold: 10MB	Limits the payload size you can send to the Eratos broker.

Buffer Overflow Behavior

Each pipeline stage for data ingestion has different behaviour when a service or user attempts to cause a buffer overflow:

Broker Receipt Overflow Behaviour:
- MQTT (QoS 0): The message is silently dropped, and the client is unaware.
- MQTT (QoS 1): The message is dropped, and the connection is terminated. The user must reconnect or rely on auto-reconnect.
- MQTT (QoS 2): Not supported by the message broker (downgraded to QoS 1 publication acknowledgment).
- AMQP: A basic.nack will be sent to the publisher, and the message is dropped (behaviour determined by reject-publish option).
Ingestion Processing Overflow Behaviour:
- All protocols: Consumption from broker message queues halts temporarily until the processing backlog clears enough to accept the next message from the broker.

Messaging Protocol Resources

Each message-based data source captures historical message parsing metrics in a child group named DATASOURCE_GROUP_NAME-metrics.

Metrics can be inspected in near real-time or via historical data. Check the live documentation.