Running a complete stack with docker-compose

To install FlexMeasures, plus the libraries and databases it depends on, on your computer is some work, and can have unexpected hurdles, e.g. depending on the operating system. A nice alternative is to let that happen within Docker. The whole stack can be run via Docker compose, saving the developer much time.

For this, we assume you are in the directory (in the FlexMeasures git repository) housing docker-compose.yml.

Note

The minimum Docker version is 17.09 and for docker-compose we tested successfully at version 1.25. You can check your versions with docker[-compose] --version.

Note

The command might also be docker compose (no dash), for instance if you are using Docker Desktop.

Build the compose stack

Run this:

$ docker-compose build

This pulls the images you need, and re-builds the FlexMeasures ones from code. If you change code, re-running this will re-build that image.

This compose script can also serve as an inspiration for using FlexMeasures in modern cloud environments (like Kubernetes). For instance, you might want to not build the FlexMeasures image from code, but simply pull the image from DockerHub.

If you wanted, you could stop building from source, and directly use the official flexmeasures image for the server and worker container (set image: lfenergy/flexmeasures in the file docker-compose.yml).

Run the compose stack

Start the stack like this:

$ docker-compose up

Warning

This might fail if ports 5000 (Flask) or 6379 (Redis) are in use on your system. Stop these processes before you continue.

Check docker ps or docker-compose ps to see if your containers are running:

$ docker ps
CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS                             PORTS                                            NAMES
beb9bf567303   flexmeasures_server   "bash -c 'flexmeasur…"   44 seconds ago   Up 38 seconds (health: starting)   0.0.0.0:5000->5000/tcp                           flexmeasures-server-1
e36cd54a7fd5   flexmeasures_worker   "flexmeasures jobs r…"   44 seconds ago   Up 5 seconds                       5000/tcp                                         flexmeasures-worker-1
c9985de27f68   postgres              "docker-entrypoint.s…"   45 seconds ago   Up 40 seconds                      5432/tcp                                         flexmeasures-test-db-1
03582d37230e   postgres              "docker-entrypoint.s…"   45 seconds ago   Up 40 seconds                      5432/tcp                                         flexmeasures-dev-db-1
25024ada1590   mailhog/mailhog       "MailHog"                45 seconds ago   Up 40 seconds                      0.0.0.0:1025->1025/tcp, 0.0.0.0:8025->8025/tcp   flexmeasures-mailhog-1
792ec3d86e71   redis                 "docker-entrypoint.s…"   45 seconds ago   Up 40 seconds                      0.0.0.0:6379->6379/tcp                           flexmeasures-queue-db-1

The FlexMeasures server container has a health check implemented, which is reflected in this output and you can see which ports are available on your machine to interact.

You can use the terminal or docker-compose logs to look at output. docker inspect <container> and docker exec -it <container> bash can be quite useful to dive into details. We’ll see the latter more in this tutorial.

Configuration

You can pass in your own configuration (e.g. for MapBox access token, or db URI, see below) like we described in Configuration and customization ― put a file flexmeasures.cfg into a local folder called flexmeasures-instance (the volume should be already mapped).

In case your configuration loads FlexMeasures plugins that have additional dependencies, you can add a requirements.txt file to the same local folder. The dependencies listed in that file will be freshly installed each time you run docker-compose up.

Data

The postgres database is a test database with toy data filled in when the flexmeasures container starts. You could also connect it to some other database (on your PC, in the cloud), by setting a different SQLALCHEMY_DATABASE_URI in the config.

Seeing it work: Running the toy tutorial

A good way to see if these containers work well together, and maybe to inspire how to use them for your own purposes, is the Toy example I: Scheduling a battery, from scratch.

The flexmeasures-server container already creates the toy account when it starts (see its initial command). We’ll now walk through the rest of the toy tutorial, with one twist at the end, when we create the battery schedule.

Let’s go into the flexmeasures-worker container:

$ docker exec -it flexmeasures-worker-1 bash

There, we’ll now add the price data, as described in Add some price data. Copy the commands from that section and run them in the container’s bash session, to create the prices and add them to the FlexMeasures DB.

Next, we put a scheduling job in the worker’s queue. This only works because we have the Redis container running ― the toy tutorial doesn’t have it. The difference is that we’re adding --as-job:

$ flexmeasures add schedule for-storage --sensor 2 --consumption-price-sensor 1 \
    --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \
    --roundtrip-efficiency 90% --as-job

We should now see in the output of docker logs flexmeasures-worker-1 something like the following:

Running Scheduling Job d3e10f6d-31d2-46c6-8308-01ede48f8fdd: discharging, from 2022-07-06 07:00:00+01:00 to 2022-07-06 19:00:00+01:00

So the job had been queued in Redis, was then picked up by the worker process, and the result should be in our SQL database container. Let’s check!

We’ll not go into the server container this time, but simply send a command:

$ TOMORROW=$(date --date="next day" '+%Y-%m-%d')
$ docker exec -it flexmeasures-server-1 bash -c "flexmeasures show beliefs --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H"

The charging/discharging schedule should be there:

┌────────────────────────────────────────────────────────────┐
│               ▐▀▀▌                                     ▛▀▀│ 0.5MW
│   ▞▌                                                    │
│   ▌▌                                                 ▗▘  │
│   ▌▌                                                    │
│                                                       │
│                ▝▖                                      │
│                                                       │
│   ▝▖                                                   │
│▀▘───▀▀▀▀▖─────▌────▀▀▀▀▀▀▀▀▀▌─────▐▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▘───│ 0.0MW
│                                                        │
│                               ▗▘                        │
│                                                        │
│                           ▝▖                            │
│                              ▗▘                         │
│                                                        │
│         ▝▖                                              │
│          ▙▄▟                 ▐▄▄▌                           -0.5MW
└────────────────────────────────────────────────────────────┘
           10           20           30          40
                        ██ discharging

Like in the original toy tutorial, we can also check in the server container’s web UI (username is “toy-user@flexmeasures.io”, password is “toy-password”):

https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/sensor-data-charging.png

Email Testing

To test email functionality, MailHog is included in the Docker Compose stack. You can view the emails sent by the application by navigating to http://localhost:8025/ in your browser.

To verify this setup, try changing a user’s password in the application. This action will trigger an email, which you can then view in MailHog.

Scripting with the Docker stack

A very important aspect of this stack is if it can be put to interesting use. For this, developers need to be able to script things ― like we just did with the toy tutorial.

Note that instead of starting a console in the containers, we can also send commands to them right away. For instance, we sent the complete flexmeasures show beliefs command and then viewed the output on our own machine. Likewise, we send the pytest command to run the unit tests (see below).

Used this way, and in combination with the powerful list of CLI Commands, this FlexMeasures Docker stack is scriptable for interesting applications and simulations!

Running tests

You can run tests in the flexmeasures docker container, using the database service test-db in the compose file (per default, we are using the dev-db database service).

After you’ve started the compose stack with docker-compose up, run:

$ docker exec -it -e SQLALCHEMY_TEST_DATABASE_URI="postgresql://fm-test-db-user:fm-test-db-pass@test-db:5432/fm-test-db" flexmeasures-server-1 pytest

This rounds up the developer experience offered by running FlexMeasures in Docker. Now you can develop FlexMeasures and also run your tests. If you develop plugins, you could extend the command being used, e.g. bash -c "cd /path/to/my/plugin && pytest".