Feed updates are off temporarily

March 3rd, 2010 | by bryan |

I’ve turned off feed updates temporarily so that I can move the spider service over to some new beefier hardware. There were several systems running on the old server and the load on this server was slowing down parts of the site. By moving this system to a new server everything should speed up a bit.

The main issue with moving this system is that ReadPath keeps lists of all of the content items that it’s already seen. There are several million lists which total 46G of files. It’s rather surprising how hard it is to move these types of files around. I did an initial rsync of the dir, which took almost an entire day since it was done while everything was still running and there was lots of random i/o on the disk. Now that I’ve turned updates off I’m doing a final rsync. This rsync is still taking a significant amount of time since even though it’s only been a couple hours since the initial rsync, a large percentage of the files have been updated with new content.

I’ll post an update as soon as I have the changes deployed and updates turned back on again.


The Feed Spider has now been turned back on again. You should see all feeds update normally now.


Post a Comment