The hard computation work (e.g. forecasting, scheduling) should happen outside of web requests (asynchronously), in job queues accessed by worker processes.
This queueing relies on a Redis server, which has to be installed locally, or used on a separate host. In the latter case, configure Redis details in your FlexMeasures config file.
Here we assume you have access to a Redis server and configured it (see Redis). The FlexMeasures unit tests use fakeredis to simulate this task queueing, with no configuration required.
See also Running a complete stack with docker-compose for usage of Redis via Docker and a more hands-on tutorial on the queues.
Here is how to run one worker for each kind of job (in separate terminals):
flexmeasures jobs run-worker --name our-only-worker --queue forecasting|scheduling
Running multiple workers in parallel might be a great idea.
flexmeasures jobs run-worker --name forecaster --queue forecasting flexmeasures jobs run-worker --name scheduler --queue scheduling
You can also clear the job queues:
flexmeasures jobs clear-queue --queue forecasting flexmeasures jobs clear-queue --queue scheduling
When the main FlexMeasures process runs (e.g. by
flexmeasures run), the queues of forecasting and scheduling jobs can be visited at
http://localhost:5000/tasks/schedules, respectively (by admins).
Inspect the queue and jobs
The first option to inspect the state of the
forecasting queue should be via the formidable RQ dashboard. If you have admin rights, you can access it at
your-flexmeasures-url/rq/, so for instance
http://localhost:5000/rq/. You can also start RQ dashboard yourself (but you need to know the redis server credentials):
pip install rq-dashboard rq-dashboard --redis-host my.ip.addr.ess --redis-password secret --redis-database 0
RQ dashboard shows you ongoing and failed jobs, and you can see the error messages of the latter, which is very useful.
Finally, you can also inspect the queue and jobs via a console (see the nice RQ documentation), which is more powerful. Here is an example of inspecting the finished jobs and their results:
from redis import Redis from rq import Queue from rq.job import Job from rq.registry import FinishedJobRegistry r = Redis("my.ip.addr.ess", port=6379, password="secret", db=2) q = Queue("forecasting", connection=r) finished = FinishedJobRegistry(queue=q) finished_job_ids = finished.get_job_ids() print("%d jobs finished successfully." % len(finished_job_ids)) job1 = Job.fetch(finished_job_ids, connection=r) print("Result of job %s: %s" % (job1.id, job1.result))
Redis queues on Windows
On Unix, the rq system is automatically set up as part of FlexMeasures’s main setup (the
However, rq is not functional on Windows without the Windows Subsystem for Linux.
On these versions of Windows, FlexMeasures’s queuing system uses an extension of Redis Queue called
This is also an automatically installed dependency of FlexMeasures.
However, the Redis server needs to be set up separately. Redis itself does not work on Windows, so it might be easiest to commission a Redis server in the cloud (e.g. on kamatera.com).
If you want to install Redis on Windows itself, it can be set up on a virtual machine as follows:
Download the vagrant-redis vagrant configuration
vagrant-redis.zipin any folder, e.g. in
config.vm.box = "hashicorp/precise64"in the Vagrantfile, and remove the line with
vagrant upin Command Prompt