flexmeasures.data.utils

Utils around the data models and db sessions

Functions

flexmeasures.data.utils.get_data_source(data_source_name: str, data_source_model: str | None = None, data_source_version: str | None = None, data_source_type: str = 'script') DataSource

Make sure we have a data source. Create one if it doesn’t exist, and add to session. Meant for scripts that may run for the first time.

flexmeasures.data.utils.save_to_db(data: BeliefsDataFrame | BeliefsSeries | list[BeliefsDataFrame | BeliefsSeries], bulk_save_objects: bool = True, save_changed_beliefs_only: bool = True) str

Save the timed beliefs to the database.

Note: This function does not commit. It does, however, flush the session. Best to keep transactions short.

We make the distinction between updating beliefs and replacing beliefs.

# Updating beliefs

An updated belief is a belief from the same source as some already saved belief, and about the same event, but with a later belief time. If it has a different event value, then it represents a changed belief. Note that it is possible to explicitly record unchanged beliefs (i.e. updated beliefs with a later belief time, but with the same event value), by setting save_changed_beliefs_only to False.

# Replacing beliefs

A replaced belief is a belief from the same source as some already saved belief, and about the same event and with the same belief time, but with a different event value. Replacing beliefs is not allowed, because messing with the history corrupts data lineage. Corrections should instead be recorded as updated beliefs. Servers in ‘play’ mode are exempt from this rule, to facilitate replaying simulations.

Parameters:
  • data – BeliefsDataFrame (or a list thereof) to be saved

  • bulk_save_objects – if True, objects are bulk saved with session.bulk_save_objects(), which is quite fast but has several caveats, see: https://docs.sqlalchemy.org/orm/persistence_techniques.html#bulk-operations-caveats

  • save_changed_beliefs_only – if True, unchanged beliefs are skipped (updated beliefs are only stored if they represent changed beliefs) if False, all updated beliefs are stored

Returns:

status string, one of the following: - ‘success’: all beliefs were saved - ‘success_with_unchanged_beliefs_skipped’: not all beliefs represented a state change - ‘success_but_nothing_new’: no beliefs represented a state change

flexmeasures.data.utils.save_to_session(objects: list[Model], overwrite: bool = False)

Utility function to save to database, either efficiently with a bulk save, or inefficiently with a merge save.