Guides
Eratos Workspace
Start typing to search…

Hussat Message Parser

Hussat Message Parser

About the parser

This parser is used to ingest data from Hussat soil moisture probes (Terrasonde) and their base stations. https://hussat.com.au/?page_id=2

This document assumes the basestations relay radio frequency Terrasonde messages to Eratos via a LoRaWAN server such as TTI (https://www.thethingsindustries.com/). It is worth noting that the messages could be coming from another telemetry source and relayed to Eratos using a MQTT Broker if the broker messages contain the necessary key attributes application_id, device_id and payload.

A brief introduction to JSONata will be useful. JSON Message Parser#JSONataQueries

Important setup conventions

Terrasonde basestations are identified by a 4 digit hex number which must be entered as the last 4 characters when configuring the LoRaWAN End Device Name. This allows data to be collected for which basestation is relaying the data for a Terrasonde message. This can help with optimising basestation placement and identifying if a Terrasonde outage or gaps in data is due to a basestation outage.

Basic parser configuration guide

The basic example will be best to follow for nearly all users especially if your messages are coming from TTI.

platform_id_infix

Used as part of the platform, stream and group ids to help you search for them.

Naming example for a stream is:
dsapi-<kodenamize_id>-<platform_id_infix>.<device_id>.terrasonde.<Terrasonde id from packet>.label

platform_id_suffix_queries

A list of JSONata queries to find a device_id in the naming example above.

sample_period_x, reporting_period_x

Standard sample and reporting periods as used during stream creation via the dashboard interface.

timestamp_queries

This config setting allows you to set the timestamp used for the incoming packet.

The first successful timestamp extraction will be used.
If no timestamp can be extracted from the message, the time of arrival is used and a warning message is logged. If you would like to use time of arrival without a warning message, set the query attribute blank (see warning below).

  • query:
    WARNING - leaving blank will result in using time of arrival. This is not recommended because if old messages are replayed such as after a system outage where queued messages are retried, the timestamp will be incorrect.
    The query is in the format of a JSONata query.

  • time_format:
    A DateTimeFormat pattern as defined at https://www.joda.org/joda-time/key_format.html
    If the parsing of time_format or time_zone fails, the timestamp_query has failed and the next query in the list is considered.
    If time_format and time_zone attributes are left out, set to null or empty string, the format will be parsed as ISO 8601.

  • time_zone:
    Canonical ID for a timezone as listed at https://www.joda.org/joda-time/timezones.html
    Only used if time_format is not left out, set to null or empty string.

lorawan_channel_plan_queries (Optional)

Not normally necessary with Hussat equipment as they should be on a fixed data rate.
As described at JSON Message Parser#lorawan_channel_plan_queries(Optional)

json_queries

As described at JSON Message Parser#json_queries(Optional)

senaps_stream_metadata

Each element of this array will be used to set metadata when streams are created for Terrasonde and Basestation platforms.
senaps_stream_metadata can still contain entries for uplink data not collected and will provision a stream to make it easier to activate at a later date.

The packet structure for the Hussat sensors is well defined and only keys as listed in the example are valid. Other keys will be ignored.

Basic configuration example

{
  "platform_id_infix": "your.stream.namespace",
  "platform_id_suffix_queries": [
    "end_device_ids.device_id"
  ],
  "timestamp_queries": [
    {
      "query": "received_at"
    },
    {
      "query": "rxInfo.time"
    }
  ],
  "data_payload_queries": [
    "uplink_message.frm_payload"
  ]
}

Initialisation checks

If the parser has successfully passed configuration validation and connected to a data source, you will see the log messages:

Hussat Parser has started and is ready for incoming messages
initialisation-check.basestationIdBase next send time at 20 minutes past the hour
Creating messages for basestation platform name: initialisation-check.basestationIdBase
Unable to calculate zeroSequenceNumberMillis for initialisation-check.terrasondeIdBase
Creating messages for basestation platform name: initialisation-check.basestationIdBase
Creating messages for terrasonde platform id: initialisation-check.terrasondeIdBase

There is no need to be concerned about the message - "Unable to calculate zeroSequenceNumberMillis" because these startup performed against a test packet that is discarded. This check detects any problems with the data source configuration during startup rather than later during operation when it is less likely to be checked after a modification.

Platforms and Streams are automatically created when a message is received from a basestation or a Terrasonde probe.

Depending on the frequency of Terrasonde data transmissions and LoRaWAN message success rate, it can sometimes take days before any data appears in the Streams. This is because the date and time the Terrasonde was turned on needs to be calculated from successive packets. If a number of message failures occur, this calculation is not possible. This may be due to LoRaWAN network capacity (message collisions) or low signal quality (long distance, geographical or physical antenna obstructions).

If you are using the option to seed_cached_terrasondes (Advanced feature), you will need to wait until the parser receives a Terrasonde or basestation message for seeding to occur and be seen in the log.

Advanced parser configuration guide

These options are either for future use, facilitate migration or ingesting from MQTT brokers other than TTI.

channel_plan (Optional)

At the moment, this option is not necessary because all messages are sent using data rate 2 however in the future, adaptive data rates may be used.

As documented at JSON Message Parser#lorawan_channel_plan_queries(Optional)

seed_version (Optional)

This config setting defines a version of seed_cached_terrasondes.

Once a version of seed_cached_terrasondes has been used to seed the terrasonde cache, it will not be used again until the next change in this config setting.

Using a text description will assist if debugging is required.

This configuration option may be left out, set to null or empty string to disable seeding.

seed_cached_terrasondes (Optional)

Finding the epoch that packet sequence ids are based on can take days if many packets are being dropped.

Seeding speeds up this process when migrating or manually initialising a Terrasonde or Basestation.

The key is the base 10 formatted numerical id for the Terrasonde as it would be extracted from the binary packet.

Leaving the value for this mapped field empty will delete any cached Terrasondes for that id which is handy if you want to ensure stale data isn't used.

This configuration option may be left out, set to null or empty string to disable seeding.

Advanced configuration example

{
  "platform_id_infix": "your.stream.namespace",
  "platform_id_suffix_queries": [
    "end_device_ids.device_id"
  ],
  "sample_period_basestation": "PT1H",
  "reporting_period_basestation": "PT1H",
  "sample_period_terrasonde": "P1D",
  "reporting_period_terrasonde": "P1D",
  "sample_period_terrasonde_short": "PT3H",
  "reporting_period_terrasonde_short": "P1D",
  "timestamp_queries": [
    {
      "query": "received_at"
    },
    {
      "query": "rxInfo.time"
    }
  ],
  "data_payload_queries": [
    "uplink_message.frm_payload"
  ],
  "lorawan_channel_plan_queries": {
    "channel_plan": "AS923",
    "label": "LORAWAN.data_rate",
    "bandwidth_queries": [
      "uplink_message.settings.data_rate.lora.bandwidth/1000",
      "txInfo.dataRate.bandwidth"
    ],
    "spreading_factor_queries": [
      "uplink_message.settings.data_rate.lora.spreading_factor",
      "txInfo.dataRate.spreadFactor"
    ],
    "sample_period": "PT2H",
    "reporting_period": "PT4H",
    "stream_metadata": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    }
  },
  "json_queries": {
    "LORAWAN.rssi": {
      "queries": [
        "uplink_message.rx_metadata[0].rssi"
      ],
      "sample_period": "PT1H",
      "reporting_period": "PT1H",
      "stream_metadata": {
        "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
        "type": ".ScalarStreamMetaData"
      }
    },
    "LORAWAN.snr": {
      "queries": [
        "uplink_message.rx_metadata[0].snr"
      ],
      "sample_period": "PT1H",
      "reporting_period": "PT1H",
      "stream_metadata": {
        "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
        "type": ".ScalarStreamMetaData"
      }
    }
  },
  "senaps_stream_metadata": {
    "seq": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "battery": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "rssi": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "rssi_noise_floor": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "dielectric_constant": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "electrical_conductivity": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "internal_temperature": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "soil_temperature": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "battery_charged_flag": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "rain_count": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/TotalPrec",
      "type": ".ScalarStreamMetaData"
    },
    "rain_count_total": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/TotalPrec",
      "type": ".ScalarStreamMetaData"
    },
    "via_basestation": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "from_terrasonde": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "seq_at_zero_sequence_set": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    },
    "epoch": {
      "interpolationType": "http://www.opengis.net/def/waterml/2.0/interpolationType/Continuous",
      "type": ".ScalarStreamMetaData"
    }
  }
}

Updated 8 days ago