Reporting

FlexMeasures feeds upon raw measurement data (e.g. solar generation) and data from third parties (e.g. weather forecasts).

However, there are use cases for enriching these raw data by combining them:

  • Pre-calculations: For example, from a tariff and some tax rules we compute the real financial impact of price data.

  • Post-calculations: To be able to show the customer value, we regularly want to compute things like money or CO₂ saved.

These calculations can be done with code, but there’ll be many repetitions.

We added an infrastructure that allows us to define computation pipelines and CLI commands for developers to list available reporters and trigger their computations regularly:

  • flexmeasures show reporters

  • flexmeasures add report

Reports can be queued for asynchronous processing with flexmeasures add report --as-job. Run flexmeasures jobs run-worker --queue reporting to process these jobs. A one-off report can also be queued through POST /api/v3_0/assets/<id>/reports/trigger. The caller needs read access to every input and configuration sensor and permission to record data on each output; outputs are limited to the asset in the URL and its descendants.

The reporter classes we are designing are using pandas under the hood and can be sub-classed, allowing us to build new reporters from stable simpler ones, and even pipelines. Remember: re-use is developer power!

We believe this infrastructure will become very powerful and enable FlexMeasures hosts and plugin developers to implement exciting new features.

Below are two quick examples, but you can also dive deeper in Toy example V: Computing reports.

Example: solar feed-in / self-consumption delta

So here is a glimpse into a reporter we made - it is based on the AggregatorReporter (which is for the combination of any two sensors). This simplified example reporter basically calculates pv - consumption at grid connection point. This tells us how much solar power we fed back to the grid (positive values) and/or the amount of grid power within the overall consumption that did not come from local solar panels (negative values).

This is the configuration of how the computation works:

{
    "method" : "sum",
    "weights" : {
        "pv" : 1.0,
        "consumption" : -1.0
    }
}

This parameterizes the computation (from which sensors does data come from, which range & where does it go):

{
    "input": [
        {
            "name" : "pv",
            "sensor": 1,
            "source" : 1,
        },
        {
            "name" : "consumption",
            "sensor": 1,
            "source" : 2,
        }
    ],
    "output": [
        {
            "sensor": 3,
        }
    ],
    "start" : "2023-01-01T00:00:00+00:00",
    "end" : "2023-01-03T00:00:00+00:00",
}

Note

In addition to filtering by specific data source IDs (source / sources), reporter input data can be filtered using:

  • source_types: list of source type names to include (e.g. ["forecaster", "scheduler"])

  • exclude_source_types: list of source type names to exclude

  • account_id: list of account IDs to include only data from sources belonging to those accounts (a data source belongs to the organisation it records for, whether a user or a forecaster, scheduler or reporter recorded the data: see Source filters)

These correspond to the same filters available on Sensor.search_beliefs.

Note

An input entry can also clean the readings it asks for, before the report is computed, with the same lower, upper and snap fields a forecaster’s regressors and a flex-config sensor reference take (see Cleaning data on the way in). This helps where a sensor records the occasional implausible value — a spike, or an error sentinel such as -9999 — that would otherwise end up in the report:

{
    "input": [{"sensor": 1, "lower": "0 kW", "snap": {"0 kW": ["0 kW", "0.1 kW"]}}]
}

Snapping runs before clipping. Bounds without a unit are read in the unit of the sensor the input names, and bounds with a unit must be convertible to it, which is checked when the report is set up rather than when its data is read.

Example: the measured aggregate of a device group

A site that is scheduled has already described its topology in its flex-config: which devices sit behind which piece of shared equipment (their group), which sensor records each of them, which sign means consumption or production, and which sensor a group’s aggregate belongs on. The AggregatorReporter can report the measured aggregate of such a group, so the report needs no topology of its own and cannot drift from the one the scheduler uses.

Take a farm whose PV sits on two installations:

Farm (site, asset 1)
└── PV (asset 2)                  sensor 21 "PV production", MW, 15 min   <- the aggregate
    ├── Roof PV (asset 3)         sensor 31 "power", kW, 15 min
    └── Carport PV (asset 4)      sensor 41 "power", MW, 1 hour

The flex-models stored on those assets are all the topology there is:

{"inflexible-production" : {"sensor" : 31}, "group" : {"asset" : 2}}

{"inflexible-production" : {"sensor" : 41}, "group" : {"asset" : 2}}

{"production" : {"sensor" : 21}}

The report’s configuration then names only the group:

{
    "group" : {"asset" : 2}
}

And its parameters only the period, with a belief horizon of zero to ask for realized values rather than forecasts:

{
    "start" : "2026-06-01T00:00:00+02:00",
    "end" : "2026-06-02T00:00:00+02:00",
    "belief_horizon" : "PT0H"
}

With the roof measuring 400 kW and the carport 0.15 MW, sensor 21 receives 0.55 MW for every quarter-hour: the roof’s kilowatts are converted to megawatts, and the carport’s hourly values are carried across each quarter of their hour. Adding a third installation means adding one flex-model entry with "group": {"asset": 2}; the scheduler and this report both pick it up, with nothing to keep in sync.

Groups of groups. A group’s entry may itself join another group, which makes the groups a tree. A report over an outer group reaches the devices behind it, not the inner groups’ own aggregate sensors:

Street (a group)                  sensor 11 "street PV", MW
└── House (a group of the street) sensor 21 "house total", MW
    ├── House PV                  sensor 31 "power", kW
    └── House heat pump           sensor 41 "power", kW

{"group" : {"asset" : <house>}}

{"group" : {"asset" : <street>}, "consumption" : {"sensor" : 21}}

{"inflexible-production" : {"sensor" : 31}, "group" : {"asset" : <house>}}

{"inflexible-consumption" : {"sensor" : 41}, "group" : {"asset" : <house>}}

Reporting the street aggregates sensors 31 and 41, with the heat pump’s consumption subtracted from the PV’s production because the street records its aggregate under production. Sensor 21 is the house’s aggregate rather than a device, so it is not read, and no value is counted twice.

Narrowing a group. A group is a piece of equipment, so it holds everything behind it. To report a category instead, filter its members:

{
    "group" : {"asset" : <street>},
    "members" : {"asset-type" : "solar"}
}

That reports all the PV on the street, reaching into each house. The asset type says what a device is, which does not change when the way it is modelled changes, so a PV installation that later becomes curtailable stays in the aggregate.

Where members may sit. A device joins a group through its flex-model entry, not through where it sits in the tree, so a group’s members are looked for anywhere below the site, exactly as a scheduler of that site counts them. A PV installation hanging directly off the site, with "group": {"asset": <street>}, is part of the street’s aggregate.

Units and resolutions. Values are converted to the unit of the output sensor at each sensor’s own resolution, before being resampled to the output’s resolution. That order matters for a conversion between a stock and a flow: a sensor recording 1 kWh every quarter of an hour is recording 4 kW, and treating its data as though it were hourly would report a quarter of the real power. Resampling then follows the quantity: an energy adds up over a longer event where a power averages over it, and a power recorded hourly holds through its hour when the output is finer. A sensor whose quantity the output sensor cannot express, such as a temperature onto a power sensor, is reported as an error rather than being added up.

Gaps. A report over a group reports an event only when every member reported it, so that a gap in one member does not quietly lower the aggregate, and it logs which sensors left the gaps. A report over sensors named in input aggregates what reported, which is what such reports have always done. skip-missing sets this either way. An event that no sensor reported is missing in both cases, because an hour nothing was measured in is not an hour in which nothing flowed. A gap is also left as a gap when resampling: a coarser sensor is carried only across its own event, never across a missing one.

A flex-config the report cannot read. Resolving a group means reading the whole site’s flex-config, so an invalid flex-model entry anywhere on the site stops the report, even on a device outside the group — a scheduler of that site would fail on it too. The error says so, and names the asset whose entry does not validate.

Example: Profits & losses

A report that should cover a use case right off the shelf for almost everyone using FlexMeasures is the ProfitOrLossReporter ― a reporter to compute how profitable your operation has been. Showing the results of your optimization is a crucial feature, and now easier than ever.

First, reporters can be stored as data sources, so they are easy to be used repeatedly and the data they generate can reference them. Our data source has ProfitOrLossReporter as model attribute and these configuration information stored on its attribute defines the reporter further (the least a ProfitOrLossReporter needs to know is a price):

{
  "data_generator": {
    "config": {
      "consumption_price_sensor": 1
    }
  }
}

And here are more excerpts from the tutorial mentioned above. Here we configure the input and output:

$ echo "
  {
      'input' : [{'sensor' : 4}],
      'output' : [{'sensor' : 9}]
  }" > profitorloss-parameters.json

The input sensor stores the power/energy flow, and the output sensor will store the report. Recall that we already provided the price sensor to use in the reporter’s data source.

$ flexmeasures add report\
  --source 6 \
  --parameters profitorloss-parameters.json \
  --start-offset DB,1D --end-offset DB,2D

Here, the ProfitOrLossReporter used as source (with Id 6) is the one we configured above. With the offsets, we control the timing ― we indicate that we want the new report to encompass the day of tomorrow (see Pandas offset strings).

The report sensor will now store all costs which we know will be made tomorrow by the schedule.

Automating reports

Instead of computing reports one at a time, you can set up an automation: a recurring task defined on an asset, which queues reporting jobs on a cron schedule. See Automations.