When running my code, ssh worked if I ssh’ed very early.
After maybe 10 seconds or so, ssh would never work again until I stopped the code (sometimes even wireless internet stopped working, I couldn’t ping anymore)
@hzeller gave me the hint to put isolcpus=3 in /boot/cmdline.txt and reboot.
Sure enough, with that setting, the library isn’t taking so many resources that the networking layer stops working.
It depends what you do. For me, with wifi and sshd, sshd becomes unresponsive without it after less than a minute, and sometimes the IP or wifi stack also hangs, making it impossible to ssh in without a reboot.
During that time, my code and display still runs
It may be wifi only, I don’t have wired ethernet to those devices.
That said, if the sshd process stops replying mid stream during the negotiation, that’s not good either.
My drive to test if it would break with ethernet, is limited given that I now have a fix.