Michiel's Blog - #pythonleidenhttps://tildeweb.nl/~michiel/tags/pythonleiden.html2026-01-22T21:00:00+01:00Blog - #pythonleidenReinout van Rees on prefect.io2026-01-22T21:00:00+01:002026-01-22T21:00:00+01:00Michiel W. Beijentag:tildeweb.nl,2026-01-22:/~michiel/reinout-python-leiden-20260122.htmlSummary of Python Leiden talk<p>Reinout is a long-time visitor of the Python Leiden meetup. In fact, he is a
regular on most nearby Python meetups and conferences. The nice thing is that
he also writes summaries of the talks he attends and he publishes those on his
<a href="https://reinout.vanrees.org/weblog/">blog</a>. I really appreciate this! Today he
gave a talk at our meetup, and I decided to ‘return the favour’ so here is my
summary of his talk!</p>
<p>Reinout discusses <a href="https://prefect.io">prefect.io</a>: it is a server that
provides a dashboard, has an API, shows workflows, and allows you to
start/restart jobs, examine logs, and such.</p>
<p>In his company much IT-related work involves loading data from web APIs,
transforming it in some form, loading it in databases and so on.</p>
<p>Previously this was all done on the command line and people had to log in over
SSH and there were constraints about rotating ssh keys which all made it very
cumbersome. Now they have a nice web UI and not so much IT restraints and it’s
much easier for people because they can just log in to a web UI and click
around.</p>
<p>In Prefect, workflows are defined. Workflows consists of one or more tasks.
Those tasks are written in python. Prefect comes with some niceties to
configure retries on your tasks, it provides a custom logging framework and a
lot more. In the UI you can manually start jobs, configure workers, and monitor
progress.</p>
<p>Prefect does not provide its own authentication mechanism at the moment. So if
you want to ‘protect’ it people are using HTTP Basic auth.</p>
<p>It comes with a docker setup, which is what Reinout is using. Also a hosted
version is available.</p>
<p>Reinout created a cookiecutter template that allows non-programmers to create
simple python tasks. This has built in workflow that creates a github
repository, and configures github actions that creates docker containers on
every commit on main, and pushes those to a docker registry.</p>
<p>He uses ‘watchtower’ in the docker compose setup, which allows him to simply
deploy the stack once, and the stack will poll for updated containers and keep
itself up to date.</p>
<p>I think it’s nice to see that they are able to provide a frictionless way to
set up, deploy and monitor data jobs for people who are more data-minded and
not hardcore sofware developers!</p>