requests is a very widely used Python package, used for making HTTP requests. As you can imagine, a project like PieFed does a lot of HTTP and so I reached for the most widely used tool for the job, thinking it would be robust and easy.
And it has been, for the most part.
some_variable = requests.get(‘https://whatever/that_thing.json’)
The thing is, unless you add a ‘timeout‘ parameter to the function call, it will wait literally forever for a response from the remote server. 99.9% of the time this will be fine and the request will either succeed or raise an exception when something goes wrong but very very occasionally your script will just hang “for no reason”.
These kinds of intermittent bugs can be very difficult to solve because the trigger (a remote server malfunctioning in a very particular way) is outside of the development environment, making it impossible to reproduce unless you already know exactly what you’re looking for. In this case I tried half a dozen other fixes before eventually giving up and writing a separate script to monitor PieFed and restart it when it got stuck.
A few weeks later a random event gave me an idea and I combed through the codebase looking for requests without timeouts, found a few and put in a timeout on each. With no way to reproduce the problem I still didn’t know if it was fixed but it’s now been about 2 weeks with no hangs so I’m pretty sure it’s solved now.
Hopefully this post finds some future Python developer and saves them a few days/weeks.
@piefedadmin also recommend adding this to your default linter config 🙂 https://docs.astral.sh/ruff/rules/request-without-timeout/
@jonny @piefedadmin I was also going to suggest linting the code base as I discovered the other day going through someone elses code that `ruff` has a rule about requiring a `timeout` on `request`.
I usually rely on PyCharm which I thought was catching lots of linter-type-things, but I see now that this is not enough to catch everything.
@piefedadmin There might be contexts where waiting until the user hits control C (in the case of an interactive command) makes sense (like if the network goes down and back up and manages to recover, although the more I think about timeouts longer than about two minutes the more I'm unsure of even that). Agreed that no timeout doesn't seem a great default.