We've hit an issue in running the new Tooz-backed worker setup. When Zookeeper has a connection issue, and the heartbeat fails with a connection error, the worker goes offline indefinitely and never recovers. It stops advertising itself as available so it never gets assigned work. Only restarting the worker process will make it recover. It should either sit idle until it can reconnect or exit the process so it can be noticed by monitoring systems and restarted.
We've hit an issue in running the new Tooz-backed worker setup. When Zookeeper has a connection issue, and the heartbeat fails with a connection error, the worker goes offline indefinitely and never recovers. It stops advertising itself as available so it never gets assigned work. Only restarting the worker process will make it recover. It should either sit idle until it can reconnect or exit the process so it can be noticed by monitoring systems and restarted.