Let's project into the world where Runnerly, as implemented previously, starts to be used by a lot of people. Features are added, bugs are fixed, and the database is steadily growing.
The first problem that we're facing is the background process that creates reports and calls Strava. Since we're having thousands of users, these tasks take most of the server resources, and users are experiencing slowdowns on the frontend.
It's getting obvious that we need to have them running on separate servers. With the monolithic application using Celery and Redis, it's not an issue. We can dedicate a couple of new servers for the background jobs.
But the biggest concern if we do this is that the Celery worker code needs to import the Flask application code to operate. So the deployment dedicated to the background workers needs to include the whole...
