Out of the box PieFed has some pretty conservative settings, intended for small (less than 10 users) to medium (less than 200 active users) instances running on small servers where memory is constrained.
Now that piefed.social is becoming busier I’ve needed to tweak a few settings:
Gunicorn
Gunicorn is the service that runs the web app within itself and accepts connections from nginx. When you get a lot of traffic sometimes gunicorn can run out of capacity to serve all the incoming requests.
Each request uses one thread so you need to have as many threads as your CPU cores can handle but not so many that they get flooded and start to context switch too much. The variables in gunicorn.conf.py called ‘processes’ and ‘threads’ determine this. The total number of threads will be processes * threads so keep ‘processes’ low and increase ‘threads’ as needed.
There will be a point where adding more threads makes things worse rather than better.
Celery
Celery is the service used to execute background tasks such as processing federation activity, sending emails and so on. If you can see the amount of RAM used by redis (used to queue up things for celery to work on) is growing larger and larger then you probably don’t have enough celery worker processes.
Bare metal celery: in /etc/default/celeryd, tweak the autoscale parameter in CELERY_OPTS. The numbers are the maximum and minimum amount of processes to run. Max should probably be no more than 2x your CPU cores.
Docker-based celery: in entrypoint_celery.sh, look for the autoscale parameter. You’ll need to rebuild the docker image after changing this.
Database connections
A default docker-based instance will be using postgresql with no tuning at all, which will not scale well. In .env.docker you can set some environment variables to give Postgresql more to work with. e.g.
POSTGRES_SHARED_BUFFERS=2GB
POSTGRES_EFFECTIVE_CACHE_SIZE=3GB
POSTGRES_WORK_MEM=8MBIn compost.yaml you will need to increase shm_size. Set shm_size to be larger than your intended shared_buffers setting. A common approach:
– shared_buffers: 25% of available RAM
– shm_size: 2x your shared_buffers value (to provide headroom)
See PGTune for ideas.
Bare-metal servers can tune their postgresql server by creating a file in /etc/postgresql/14/main/conf.d/whatever.conf.
Ensure your postgresql server can handle 300 connections – it’s not just each gunicorn thread that will be connecting, there will be celery processes too.
Here is the piefed.conf I’m using on piefed.social with ~1000 MAU, 8 CPU cores and 16 GB of RAM:
synchronous_commit = off
wal_writer_delay = 2000ms
max_connections = 300
shared_buffers = 6GB
effective_cache_size = 9GB
maintenance_work_mem = 1GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 10MB
huge_pages = off
min_wal_size = 1GB
max_wal_size = 3GB
wal_compression = on
max_worker_processes = 8
max_parallel_workers_per_gather = 3
max_parallel_workers = 8
max_parallel_maintenance_workers = 4
max_worker_processes should be the number of CPU cores you have.
The main web app container / service has two environment variables that control the database connection pool and the defaults (used when no value is in the .env) are quite small. Try:
DB_MAX_OVERFLOW = 70
DB_POOL_SIZE = 30
Scrapers and bots
Sooner or later they’ll find your instance and start trying to download every post so fast that your instance starts to be overloaded. In the admin UI at /admin/misc, turn on the “Private instance – require login to browse” setting. Consider anubis as a longer-term solution.
Don’t panic
This is an evolving situation and there is no one size fits all configuration. Drop in to chat.piefed.social to share what you find or ask for help.