Data Forward FAQ
Frequently asked questions concerning data forwards
Julien Stamatakis avatar
Written by Julien Stamatakis
Updated over a week ago

What is a data forward?

A data forward is a way to stream near real-time sensor data to your own cloud, server, or third-party analytics provider. Senseware will periodically make HTTP POST requests to a URL of your choosing and push the latest sensor data to this URL.

Can data forwards be created automatically?

Currently we do not provide an automated way to create data forwards through an API. Data forwards must be created through the UI. However, this is a frequently requested feature and is in the product road-map.

What are the security recommendations?

We recommend only streaming data via https. This ensures that data between Senseware and your platform remains encrypted while it is in transit. Additionally we recommend adding a random authentication token in the URL parameters. This token serves as a pre-shared key (PSK) between Senseware and your platform and allows you to verify that it is really Senseware who is pushing the data.

An example of a secure URL looks like this:

In the future Senseware may support other authentication mechanisms such as HTTP Simple Auth and other token-based systems.

What's the difference between polling frequency and publishing frequency?

The polling frequency defines the frequency at which a node collects data. E.g. a polling frequency of 15 minutes means that a point of data is created for each channel of each sensor connected to the node every 15 minutes.

The publishing frequency defines how often data is published to your platform. It is independent of the polling frequency. If the publishing frequency is greater (i.e. less frequent) than the polling frequency Senseware will publish "micro-batches", i.e. multiple reading will be sent to your platform in a single request. If the polling frequency is less than or equal to the publishing frequency, Senseware will usually publish one point of data with each request. 

How many points of data are published in each request?

The calculation for the typical batch size is as follows:
If the publishing frequency is greater than (i.e. more frequent) or equal to the polling frequency the typical batch size is one.
If the publishing frequency is less frequent than the polling frequency the typical batch size is the publishing frequency divided by the polling frequency.

Polling Frequency: 15 minutes
Publishing Frequency: 1 hour
Typical batch size: 60 minutes / 15 minutes = 4 points of data per request

Does this mean that I can expect the number of packets to be published to be always the same?

No. There are two common cases where Senseware publishes more than the typical batch size.

  1. If the site at which Senseware is installed loses internet connectivity (e.g. due to an outage caused by the ISP), the Senseware Gateway will store the collected sensor data locally. As soon as the internet connectivity is restored, the Gateway will begin to upload stored data. 

  2. For very large installations, or if there are changing physical conditions at the installation site nodes may decide to change the route to the gateway because a more optimal route was found. This can cause a slight delay in data which could cause it to "miss" its usual publishing interval and that specific point of data would then be included in the next request.

In general, we will publish all data that has reached our servers. So if an installation is down for multiple hours and comes back online, all the data that was collected during the down time that has been uploaded to the Senseware cloud will be published with the next request to your platform.

Is data always collected and published on the hour?

No. A node will immediately start to collect data when it is plugged in. It will then collect data in regular intervals (the polling frequency) but the time of the collection will not necessarily be "round".

The same applies to data forwards. Data will be published at regular intervals (the publishing frequency), but the time of publishing will not necessarily be "round".

How does Senseware handle HTTP POST failures?

Senseware will keep up to 1000 points of data for each channel "buffered" in our cloud. If your platform does either not respond within 3 seconds to our HTTP POST, or responds with a HTTP 500 Server Error, we will not delete published points of data from our buffer. We will then attempt to publish these points of data again when the next publishing interval rolls around.

This is for your convenience. If your platform is unvailable for any reason for a reasonably short amount of time, you will not miss any data. However, it is your responsibility to handle HTTP Requests properly (e.g., by responding with HTTP 200 when data was successfully received or HTTP 500 when an error occurred, database transaction, etc.) so that you don't receive duplicate data. 

What if my server is down for a long period of time? Can I still get historic data?

For bulk historic data access we offer FTP access. Please see the FTP Guide for more information.

Did this answer your question?