FlexMeasures Changelog
v0.13.0 | May 1, 2023
Warning
Sunset notice for API versions 1.0, 1.1, 1.2, 1.3 and 2.0: after upgrading to flexmeasures==0.13, users of these API versions may receive HTTP status 410 (Gone) responses.
See the documentation for deprecation and sunset.
The relevant endpoints have been deprecated since flexmeasures==0.12.
Warning
The API endpoint ([POST] /sensors/(id)/schedules/trigger) to make new schedules sunsets the deprecated (since v0.12) storage flexibility parameters (they move to the flex-model parameter group), as well as the parameters describing other sensors (they move to flex-context).
Warning
Upgrading to this version requires running flexmeasures db upgrade (you can create a backup first with flexmeasures db-ops dump).
Note
Read more on these features on the FlexMeasures blog.
New features
Keyboard control over replay [see PR #562]
Overlay charts (e.g. power profiles) on the asset page using the sensors_to_show attribute, and distinguish plots by source (different trace), sensor (different color) and source type (different stroke dash) [see PR #534]
The
FLEXMEASURES_MAX_PLANNING_HORIZONconfig setting can also be set as an integer number of planning steps rather than just as a fixed duration, which makes it possible to schedule further ahead in coarser time steps [see PR #583]Different text styles for CLI output for errors, warnings or success messages. [see PR #609]
Added API endpoints and webpages /accounts and /accounts/<id> to list accounts and show an overview of the assets, users and account roles of an account [see PR #605]
Avoid redundantly recomputing jobs that are triggered without a relevant state change. FLEXMEASURES_JOB_CACHE_TTL config setting defines the time in which the jobs with the same arguments are not being recomputed. [see PR #616]
Bugfixes
Infrastructure / Support
Support blackout tests for sunset API versions [see PR #651]
Sunset API versions 1.0, 1.1, 1.2, 1.3 and 2.0 [see PR #650]
Sunset several API fields for /sensors/<id>/schedules/trigger (POST) that have moved into the
flex-modelorflex-contextfields [see PR #580]Fix broken make show-data-model command [see PR #638]
Bash script for a clean database to run toy-tutorial by using make clean-db db_name=database_name command [see PR #640]
v0.12.3 | February 28, 2023
Bugfixes
Fix premature deserialization of
flex-contextfield for /sensors/<id>/schedules/trigger (POST) [see PR #593]
v0.12.2 | February 4, 2023
Bugfixes
Fix CLI command
flexmeasures schedule for-storagewithout--as-jobflag [see PR #589]
v0.12.1 | January 12, 2023
Bugfixes
Fix validation of (deprecated) API parameter
roundtrip-efficiency[see PR #582]
v0.12.0 | January 4, 2023
Warning
After upgrading to flexmeasures==0.12, users of API versions 1.0, 1.1, 1.2, 1.3 and 2.0 will receive "Deprecation" and "Sunset" response headers, and warnings are logged for FlexMeasures hosts whenever users call API endpoints in these deprecated API versions.
The relevant endpoints are planned to become unresponsive in flexmeasures==0.13.
Warning
Upgrading to this version requires running flexmeasures db upgrade (you can create a backup first with flexmeasures db-ops dump).
Note
Read more on these features on the FlexMeasures blog.
New features
Hit the replay button to visually replay what happened, available on the sensor and asset pages [see PR #463 and PR #560]
Ability to provide your own custom scheduling function [see PR #505]
Visually distinguish forecasts/schedules (dashed lines) from measurements (solid lines), and expand the tooltip with timing info regarding the forecast/schedule horizon or measurement lag [see PR #503]
The asset page also allows to show sensor data from other assets that belong to the same account [see PR #500]
The CLI command
flexmeasures monitor latest-loginsupports to check if (bot) users who are expected to contact FlexMeasures regularly (e.g. to send data) fail to do so [see PR #541]The CLI command
flexmeasures show beliefssupports showing beliefs data in a custom resolution and/or timezone, and also saving the shown beliefs data to a CSV file [see PR #519]Improved import of time series data from CSV file: 1) drop duplicate records with warning, 2) allow configuring which column contains explicit recording times for each data point (use case: import forecasts) [see PR #501], 3) localize timezone naive data, 4) support reading in datetime and timedelta values, 5) remove rows with NaN values, and 6) filter by values in specific columns [see PR #521]
Filter data by source in the API endpoint /sensors/data (GET) [see PR #543]
Allow posting
nullvalues to /sensors/data (POST) to correctly space time series that include missing values (the missing values are not stored) [see PR #549]Allow setting a custom planning horizon when calling /sensors/<id>/schedules/trigger (POST), using the new
durationfield [see PR #568]New resampling functionality for instantaneous sensor data: 1)
flexmeasures show beliefscan now handle showing (and saving) instantaneous sensor data and non-instantaneous sensor data together, and 2) the API endpoint /sensors/data (GET) now allows fetching instantaneous sensor data in a custom frequency, by using the “resolution” field [see PR #542]
Bugfixes
The CLI command
flexmeasures show beliefsnow supports plotting time series data that includes NaN values, and provides better support for plotting multiple sensors that do not share the same unit [see PR #516 and PR #539]Fixed JSON wrapping of return message for /sensors/data (GET) [see PR #543]
Consistent CLI/UI support for asset lat/lng positions up to 7 decimal places (previously the UI rounded to 4 decimal places, whereas the CLI allowed more than 4) [see PR #522]
Stop trimming the planning window in response to price availability, which is a problem when SoC targets occur outside of the available price window, by making a simplistic assumption about future prices [see PR #538]
Faster loading of initial charts and calendar date selection [see PR #533]
Infrastructure / Support
Reduce size of Docker image (from 2GB to 1.4GB) [see PR #512]
Allow extra requirements to be freshly installed when running
docker-compose up[see PR #528]Remove bokeh dependency and obsolete UI views [see PR #476]
Fix
flexmeasures db-ops dumpandflexmeasures db-ops restorenot working in docker containers [see PR #530] and incorrectly reporting a success when pg_dump and pg_restore are not installed [see PR #526]Plugins can save BeliefsSeries, too, instead of just BeliefsDataFrames [see PR #523]
Improve documentation and code w.r.t. storage flexibility modelling ― prepare for handling other schedulers & merge battery and car charging schedulers [see PR #511, PR #537 and PR #566]
Revised strategy for removing unchanged beliefs when saving data: retain the oldest measurement (ex-post belief), too [see PR #518]
Scheduling test for maximizing self-consumption, and improved time series db queries for fixed tariffs (and other long-term constants) [see PR #532]
Clean up table formatting for
flexmeasures showCLI commands [see PR #540]Add
"Deprecation"and"Sunset"response headers for API users of deprecated API versions, and log warnings for FlexMeasures hosts when users still use them [see PR #554 and PR #565]Explain how to avoid potential
SMTPRecipientsRefusederrors when using FlexMeasures in combination with a mail server [see PR #558]Set a limit to the allowed planning window for API users, using the
FLEXMEASURES_MAX_PLANNING_HORIZONsetting [see PR #568]
Warning
The API endpoint ([POST] /sensors/(id)/schedules/trigger) to make new schedules will (in v0.13) sunset the storage flexibility parameters (they move to the flex-model parameter group), as well as the parameters describing other sensors (they move to flex-context).
Warning
The CLI command flexmeasures monitor tasks has been deprecated (it’s being renamed to flexmeasures monitor last-run). The old name will be sunset in version 0.13.
Warning
The CLI command flexmeasures add schedule has been renamed to flexmeasures add schedule for-storage. The old name will be sunset in version 0.13.
v0.11.3 | November 2, 2022
Bugfixes
Fix scheduling with imperfect efficiencies, which resulted in exceeding the device’s lower SoC limit. [see PR #520]
Fix scheduler for Charge Points when taking into account inflexible devices [see PR #517]
Prevent rounding asset lat/long positions to 4 decimal places when editing an asset in the UI [see PR #522]
v0.11.2 | September 6, 2022
Bugfixes
v0.11.1 | September 5, 2022
Bugfixes
v0.11.0 | August 28, 2022
New features
The asset page now shows the most relevant sensor data for the asset [see PR #449]
Individual sensor charts show available annotations [see PR #428]
New API options to further customize the optimization context for scheduling, including the ability to use different prices for consumption and production (feed-in) [see PR #451]
Admins can group assets by account on dashboard & assets page [see PR #461]
Collapsible side-panel (hover/swipe) used for date selection on sensor charts, and various styling improvements [see PR #447 and PR #448]
Add CLI command
flexmeasures jobs show-queues[see PR #455]Switched from 12-hour AM/PM to 24-hour clock notation for time series chart axis labels [see PR #446]
Get data in a given resolution [see PR #458]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Do not fail asset page if entity addresses cannot be built [see PR #457]
Asynchronous reloading of a chart’s dataset relies on that chart already having been embedded [see PR #472]
Time scale axes in sensor data charts now match the requested date range, rather than stopping at the edge of the available data [see PR #449]
The docker-based tutorial now works with UI on all platforms (port 5000 did not expose on MacOS) [see PR #465]
Fix interpretation of scheduling results in toy tutorial [see PR #466 and PR #475]
Avoid formatting datetime.timedelta durations as nominal ISO durations [see PR #459]
Account admins cannot add assets to other accounts any more; and they are shown a button for asset creation in UI [see PR #488]
Infrastructure / Support
Docker compose stack now with Redis worker queue [see PR #455]
Allow access tokens to be passed as env vars as well [see PR #443]
Queue workers can get initialised without a custom name and name collisions are handled [see PR #455]
New API endpoint to get public assets [see PR #461]
Allow editing an asset’s JSON attributes through the UI [see PR #474]
Allow a custom message when monitoring latest run of tasks [see PR #489]
v0.10.1 | August 12, 2022
Bugfixes
Fix some UI styling regressions in e.g. color contrast and hover effects [see PR #441]
v0.10.0 | May 8, 2022
New features
New design for FlexMeasures’ UI back office [see PR #425]
Improve legibility of chart axes [see PR #413]
API provides health readiness check at /api/v3_0/health/ready [see PR #416]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Fix small problems in support for the admin-reader role & role-based authorization [see PR #422]
Infrastructure / Support
Dockerfile to run FlexMeasures in container; also docker-compose file [see PR #416]
Unit conversion prefers shorter units in general [see PR #415]
Shorter CI builds in Github Actions by caching Python environment [see PR #361]
Allow to filter data by source using a tuple instead of a list [see PR #421]
v0.9.4 | April 28, 2022
Bugfixes
Support checking validity of custom units (i.e. non-SI, non-currency units) [see PR #424]
v0.9.3 | April 15, 2022
Bugfixes
Let registered plugins use CLI authorization [see PR #411]
v0.9.2 | April 10, 2022
Bugfixes
v0.9.1 | March 31, 2022
Bugfixes
v0.9.0 | March 25, 2022
Warning
Upgrading to this version requires running flexmeasures db upgrade (you can create a backup first with flexmeasures db-ops dump).
New features
Three new CLI commands for cleaning up your database: delete 1) unchanged beliefs, 2) NaN values or 3) a sensor and all of its time series data [see PR #328]
Add CLI option to pass a data unit when reading in time series data from CSV, so data can automatically be converted to the sensor unit [see PR #341]
Add CLI option to specify custom strings that should be interpreted as NaN values when reading in time series data from CSV [see PR #357]
Add CLI commands
flexmeasures add sensor,flexmeasures add asset-type,flexmeasures add beliefs(which were experimental features before) [see PR #337]Add CLI commands for showing organisational structure [see PR #339]
Add CLI command for showing time series data [see PR #379]
Add CLI command for attaching annotations to assets:
flexmeasures add holidaysadds public holidays [see PR #343]Add CLI command for resampling existing sensor data to new resolution [see PR #360]
Add CLI command to delete an asset, with its sensors and data. [see PR #395]
Add CLI command to edit/add an attribute on an asset or sensor. [see PR #380]
Add CLI command to add a toy account for tutorials and trying things [see PR #368]
Add CLI command to create a charging schedule [see PR #372]
Support for percent (%) and permille (‰) sensor units [see PR #359]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Infrastructure / Support
Plugins can import common FlexMeasures classes (like
AssetandSensor) from a central place, usingfrom flexmeasures import Asset, Sensor[see PR #354]Adapt CLI command for entering some initial structure (
flexmeasures add structure) to new datamodel [see PR #349]Align documentation requirements with pip-tools [see PR #384]
Beginning API v3.0 - more REST-like, supporting assets, users and sensor data [see PR #390 and PR #392]
v0.8.0 | January 24, 2022
Warning
Upgrading to this version requires running flexmeasures db upgrade (you can create a backup first with flexmeasures db-ops dump).
Warning
In case you use FlexMeasures for simulations using FLEXMEASURES_MODE = "play", allowing to overwrite data is now set separately using FLEXMEASURES_ALLOW_DATA_OVERWRITE. Add FLEXMEASURES_ALLOW_DATA_OVERWRITE = True to your config settings to keep the old behaviour.
Note
v0.8.0 is doing much of the work we need to do to move to the new data model (see A note on the ongoing data model transition). We hope to keep the migration steps for users very limited. One thing you’ll notice is that we are copying over existing data to the new model (which will be kept in sync) with the db upgrade command (see warning above), which can take a few minutes.
New features
Bar charts of sensor data for individual sensors, that can be navigated using a calendar [see PR #99 and PR #290]
Charts with sensor data can be requested in one of the supported [vega-lite themes] (incl. a dark theme) [see PR #221]
Mobile friendly (responsive) charts of sensor data, and such charts can be requested with a custom width and height [see PR #313]
Schedulers take into account round-trip efficiency if set [see PR #291]
Schedulers take into account min/max state of charge if set [see PR #325]
Fallback policies for charging schedules of batteries and Charge Points, in cases where the solver is presented with an infeasible problem [see PR #267 and PR #270]
Note
Read more on these features on the FlexMeasures blog.
Deprecations
The Portfolio and Analytics views are deprecated [see PR #321]
Bugfixes
Fix recording time of schedules triggered by UDI events [see PR #300]
Set bar width of bar charts based on sensor resolution [see PR #310]
Fix bug in sensor data charts where data from multiple sources would be stacked, which incorrectly suggested that the data should be summed, whereas the data represents alternative beliefs [see PR #228]
Infrastructure / Support
Account-based authorization, incl. new decorators for endpoints [see PR #210]
Central authorization policy which lets database models codify who can do what (permission-based) and relieve API endpoints from this [see PR #234]
Improve data specification for forecasting models using timely-beliefs data [see PR #154]
Properly attribute Mapbox and OpenStreetMap [see PR #292]
Allow plugins to register their custom config settings, so that FlexMeasures can check whether they are set up correctly [see PR #230 and PR #237]
Add sensor method to obtain just its latest state (excl. forecasts) [see PR #235]
Migrate attributes of assets, markets and weather sensors to our new sensor model [see PR #254 and project 9]
Migrate all time series data to our new sensor data model based on the timely beliefs lib [see PR #286 and project 9]
Support the new asset model (which describes the organisational structure, rather than sensors and data) in UI and API. Until the transition to our new data model is completed, the new API for assets is at /api/dev/generic_assets. [see PR #251 and PR #290]
Internal search methods return most recent beliefs by default, also for charts, which can make them load a lot faster [see PR #307 and PR #312]
Support unit conversion for posting sensor data [see PR #283 and PR #293]
Improve the core device scheduler to support dealing with asymmetric efficiency losses of individual devices, and with asymmetric up and down prices for deviating from previous commitments (such as a different feed-in tariff) [see PR #291]
Stop automatically triggering forecasting jobs when API calls save nothing new to the database, thereby saving redundant computation [see PR #303]
v0.7.1 | November 8, 2021
Bugfixes
Fix device messages, which were mixing up older and more recent schedules [see PR #231]
v0.7.0 | October 26, 2021
Warning
Upgrading to this version requires running flexmeasures db upgrade (you can create a backup first with flexmeasures db-ops dump).
Warning
The config setting FLEXMEASURES_PLUGIN_PATHS has been renamed to FLEXMEASURES_PLUGINS. The old name still works but is deprecated.
New features
Set a logo for the top left corner with the new FLEXMEASURES_MENU_LOGO_PATH setting [see PR #184]
Add an extra style-sheet which applies to all pages with the new FLEXMEASURES_EXTRA_CSS_PATH setting [see PR #185]
Data sources can be further distinguished by what model (and version) they ran [see PR #215]
Enable plugins to automate tests with app context [see PR #220]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Infrastructure / Support
FlexMeasures plugins can be Python packages now. We provide a cookie-cutter template for this approach. [see PR #182]
Set default timezone for new users using the FLEXMEASURES_TIMEZONE config setting [see PR #190]
To avoid databases from filling up with irrelevant information, only beliefs data representing changed beliefs are saved, and unchanged beliefs are dropped [see PR #194]
Monitored CLI tasks can get better names for identification [see PR #193]
Less custom logfile location, document logging for devs [see PR #196]
Keep forecasting and scheduling jobs in the queues for only up to one day [see PR #198]
v0.6.1 | October 23, 2021
New features
Bugfixes
Infrastructure / Support
v0.6.0 | September 3, 2021
Warning
Upgrading to this version requires running flexmeasures db upgrade (you can create a backup first with flexmeasures db-ops dump).
In case you are using experimental developer features and have previously set up sensors, be sure to check out the upgrade instructions in PR #157. Furthermore, if you want to create custom user/account relationships while upgrading (otherwise the upgrade script creates accounts based on email domains), check out the upgrade instructions in PR #159. If you want to use both of these custom upgrade features, do the upgrade in two steps. First, as described in PR 157 and upgrading up to revision b6d49ed7cceb, then as described in PR 159 for the rest.
Warning
The config setting FLEXMEASURES_LISTED_VIEWS has been renamed to FLEXMEASURES_MENU_LISTED_VIEWS.
Warning
Plugins now need to set their version on their module rather than on their blueprint. See the documentation for writing plugins.
New features
Multi-tenancy: Supporting multiple customers per FlexMeasures server, by introducing the Account concept. Accounts have users and assets associated. [see PR #159 and PR #163]
In the UI, the root view (“/”), the platform name and the visible menu items can now be more tightly controlled (per account roles of the current user) [see also PR #163]
Analytics view offers grouping of all assets by location [see PR #148]
Add (experimental) endpoint to post sensor data for any sensor. Also supports our ongoing integration with data internally represented using the timely beliefs lib [see PR #147]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Infrastructure / Support
Add possibility to send errors to Sentry [see PR #143]
Add CLI task to monitor if tasks ran successfully and recently enough [see PR #146]
Document how to use a custom favicon in plugins [see PR #152]
Allow plugins to register multiple Flask blueprints [see PR #171]
Continue experimental integration with timely beliefs lib: link multiple sensors to a single asset [see PR #157]
The experimental parts of the data model can now be visualised, as well, via make show-data-model (add the –dev option in Makefile) [also in PR #157]
v0.5.0 | June 7, 2021
Warning
If you retrieve weather forecasts through FlexMeasures: we had to switch to OpenWeatherMap, as Dark Sky is closing. This requires an update to config variables ― the new setting is called OPENWEATHERMAP_API_KEY.
New features
Allow plugins to overwrite UI routes and customise the teaser on the login form [see PR #106]
Allow plugins to customise the copyright notice and credits in the UI footer [see PR #123]
Display loaded plugins in footer and support plugin versioning [see PR #139]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Infrastructure / Support
Add tutorials on how to add and read data from FlexMeasures via its API [see PR #130]
For weather forecasts, switch from Dark Sky (closed from Aug 1, 2021) to OpenWeatherMap API [see PR #113]
Entity address improvements: add new id-based fm1 scheme, better documentation and more validation support of entity addresses [see PR #81]
Re-use the database between automated tests, if possible. This shaves 2/3rd off of the time it takes for the FlexMeasures test suite to run [see PR #115]
Make assets use MW as their default unit and enforce that in CLI, as well (API already did) [see PR #108]
Let CLI package and plugins use Marshmallow Field definitions [see PR #125]
add time_utils.get_recent_clock_time_window() function [see PR #135]
v0.4.1 | May 7, 2021
Bugfixes
v0.4.0 | April 29, 2021
Warning
Upgrading to this version requires running flexmeasures db upgrade (you can create a backup first with flexmeasures db-ops dump).
New features
Allow for views and CLI functions to come from plugins [see also PR #91]
Configure the UI menu with
FLEXMEASURES_LISTED_VIEWS[see PR #91]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Asset edit form displayed wrong error message. Also enabled the asset edit form to display the invalid user input back to the user [see PR #93]
Infrastructure / Support
Updated dependencies, including Flask-Security-Too [see PR #82]
Improved documentation after user feedback [see PR #97]
Begin experimental integration with timely beliefs lib: Sensor data as TimedBeliefs [see PR #79 and PR #99]
Add sensors with CLI command currently meant for developers only [see PR #83]
Add data (beliefs about sensor events) with CLI command currently meant for developers only [see PR #85 and PR #103]
v0.3.1 | April 9, 2021
Bugfixes
PostMeterData endpoint was broken in API v2.0 [see PR #95]
v0.3.0 | April 2, 2021
New features
FlexMeasures can be installed with
pipand its CLI commands can be run withflexmeasures[see PR #54]Optionally setting recording time when posting data [see PR #41]
Add assets and weather sensors with CLI commands [see PR #74]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Show screenshots in documentation and add some missing content [see PR #60]
Documentation listed 2.0 API endpoints twice [see PR #59]
Better xrange and title if only schedules are plotted [see PR #67]
User page did not list number of assets correctly [see PR #64]
Missing postPrognosis endpoint for >1.0 API blueprints [part of PR #41]
Infrastructure / Support
Added concept pages to documentation [see PR #65]
Dump and restore postgres database as CLI commands [see PR #68]
Improved installation tutorial as part of [PR #54]
Moved developer docs from Readmes into the main documentation [see PR #73]
Ensured unique sensor ids for all sensors [see PR #70 and (fix) PR #77]
v0.2.3 | February 27, 2021
New features
Power charts available via the API [see PR #39]
User management via the API [see PR #25]
Better visibility of asset icons on maps [see PR #30]
Note
Read more on these features on the FlexMeasures blog.
Bugfixes
Infrastructure / Support
CI via Github Actions [see PR #1]
Integration with timely beliefs lib: Sensors [see PR #13]
Apache 2.0 license [see PR #16]
Load js & css from CDN [see PR #21]
Start using marshmallow for input validation, also introducing
HTTP status 422 (Unprocessable Entity)in the API [see PR #25]Replace
solarpywithpvlib(due to license conflict) [see PR #16]Stop supporting the creation of new users on asset creation (to reduce complexity) [see PR #36]