I thought I had it when I switched the checked URL to www, it stopped receiving the 500 for some time, but now after 20 minutes it started returning 500 as well.
Could this all be part of DDOS defense? I'm pinging the same 5 URLs on the checked site with interval of 10 seconds. But that wouldn't explain why 4 other URLs return 200, and no problem in browser.