Data pipelining

FlexMeasures reads time-series data in many places: a forecaster trains on it, a scheduler takes its constraints and prices from it, and a reporter computes from it. This page describes how such data is referenced, how it can be cleaned on the way in, and how it can be bounded on the way out.

Referencing time-series data

Wherever a field accepts the data of a sensor, it accepts a sensor reference: a small object that names the sensor, and can say more about which of its data to read.

{"sensor": 55, "source-types": ["forecaster"], "default": "15 kW"}

References are accepted in:

  • the fields of the flex-model and the flex-context, as one of the forms of a variable quantity (see Variable quantities),

  • the config of a forecaster, for the sensors it reads (see Forecasting),

  • the input of a reporter (see Reporting).

In a forecaster’s config, a bare sensor ID works too.

Source filters

Several sources (for example a forecast and a schedule) can record beliefs on the same sensor. A source filter keeps the reference pointing at the right data:

  • source-types / exclude-source-types: include or exclude sources by type (e.g. "forecaster", "scheduler", "user"). Recommended over a specific source ID, because forecasters and schedulers are versioned: a version bump gives new data a new source ID, but the source type stays the same, so filters based on it don’t need updating.

  • source-account: a list of organisation IDs, to filter by the organisation(s) linked to data sources. A data source belongs to the organisation it records for: for data recorded by a user, the organisation of that user, and for data generated by a forecaster, scheduler or reporter, the organisation it computes for.

  • sources: a list of specific data source IDs. Pass a single-element list to filter on one source. When two listed sources hold beliefs about the same event, recorded at the same time, the first listed source wins.

Each of these keys takes a list, also where it names a single source. A reference refuses a key it does not take, so that a filter that is misspelled fails, rather than quietly dropping the filtering it was meant to apply.

The same filters exist elsewhere, with a different spelling:

Where

Spelling

Sensor reference (flex-model, flex-context, forecaster config)

source-types, exclude-source-types, source-account, sources (all lists)

Input of a reporter

snake case: source_types, exclude_source_types, source_account_ids, sources (see Reporting)

Query of a data request

singular: source, source-type, source-account (see Sources)

Fallback values

A reference in the flex-model or the flex-context can also name a default: a fallback quantity to use for the moments where the sensor holds no value. How it fills a sparse sensor, and why a fallback of 0 on a capacity needs care, is described in Sensor references.

Cleaning data on the way in

A sensor may record the occasional implausible value, such as a negative reading from a meter that can only measure consumption, or an error sentinel such as -9999. A sensor reference can clean the readings it points at before they are used, with the same three fields everywhere:

{"sensor": 58, "lower": "0 kW", "snap": {"0 kW": ["0 kW", "0.1 kW"]}}

  • lower / upper: readings below or above these values are clipped to them.

  • snap: a mapping from a target value to an interval; readings inside the interval are replaced by the target.

The rules are the same wherever the fields are used:

  • Snapping runs first and clipping runs afterwards, so lower and upper always take precedence: a snap target outside the bounds is clipped back into range.

  • The first bound of a snap interval is inclusive and the second is exclusive, so an interval reads as [first, second). Listing the bounds in reverse order flips the closed side: ["10 kW", "4 kW"] means (4 kW, 10 kW]. This keeps adjacent intervals unambiguous: a value on a shared boundary belongs to the interval that opens at it.

  • The snap target must lie within its interval (on a bound or inside it), so values never snap to a value outside the interval.

  • Bounds without a unit are read in the unit of the sensor they clean, and bounds with a unit must be convertible to it.

  • Values missing from the sensor are left missing.

Cleaning applies at these places. Bounds the sensor cannot take, such as a unit that does not convert, are refused before any data is read:

Data generator

Where the bounds go, and when cleaning runs

Forecaster

On a sensor reference in the forecaster config, for a regressor or for the sensor being forecast. Cleaning runs after missing values are filled, so a value interpolated across a gap is bounded too. See Cleaning the data a forecaster trains on.

Scheduler

On a sensor reference in the flex-model or the flex-context. Cleaning runs before the readings are resampled, converted or defaulted, so a missing reading is still left to the default of the reference. See Cleaning a referenced sensor’s readings.

Reporter

On an entry of the input. Cleaning runs before the report is computed. See Reporting.

Out-of-range readings are clipped to the bound, not treated as missing, so a forecaster does not interpolate across them and a scheduler does not fall back to the default. Where readings are wrong rather than merely out of range, correcting them at the source is still the better fix.

Bounding data on the way out

A forecaster can also bound what it writes: lower, upper and snap in its own config clip and snap the forecasts before they are stored, following the same rules as above, with unitless values read in the unit of the output sensor. See Forecasting.

Schedulers and reporters take no such bounds. The limits on a schedule belong in the flex-model (see The flex-models & corresponding schedulers).

Where to read more