Anyone know a good web spider/crawler? I want something which can start with a given URL and retrieve pages to a specified depth. Search capability with support for special characters is a bonus.
I'm rusty on the details because it's been a while since I set it up, but I'm pretty sure it can do what you want -- for example, IIRC, the site search on An Tir Heralds searches both antirheralds.org and the Internal Letter archive on Badger's server. (There is also a search that can be done that only searches the ILs, but I am pretty sure I set up the main search to do both servers.)
I would go check on it to refresh my memory on how it's set up, but I have to leave the house right now.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
(no subject)
Date: 2006-05-01 01:35 am (UTC)I would go check on it to refresh my memory on how it's set up, but I have to leave the house right now.