(no subject)

[ursula]

Anyone know a good web spider/crawler? I want something which can start with a given URL and retrieve pages to a specified depth. Search capability with support for special characters is a bonus.

[Same game.]

Comments

[dymaxion.livejournal.com]

Fewer em dashes than I tend to use, and no non-breaking space between the sentences, but that's probably me, certainly by my more recent posts.

wget should do what you need.

[ursula]

It is. (You're overestimating your dash use, I think. My unadjusted prose style is far more punctuation-happy than yours, at least judging by livejournal.)

[dymaxion.livejournal.com]

Hrm; I may be—I've been been getting into semicolons lately, and trying to moderate my em dash use and write in sentences of less than paragraph length.  While those work well enough for academic papers, they aren't as great in the real world.

[dymaxion.livejournal.com]

(Not to impugn your writing, of course—I'm only speaking about personal experience)

[ursula]

(I should note that the application I have in mind is a quick-and-dirty search process for the MNA which could be implemented by people who aren't particularly computer-geek-oriented, so though wget does what I asked for, the way it does it may not be ideal.)

[wengeue.livejournal.com]

This (http://www.xav.com/scripts/search/) is what the An Tir Heralds site uses. I don't know how it deals with special characters, but it's worth checking.

[ursula]

Do you know how it deals with pages on different servers? The big problem with searching the MNA is that lots of the pages aren't locally hosted.

[wengeue.livejournal.com]

I'm rusty on the details because it's been a while since I set it up, but I'm pretty sure it can do what you want -- for example, IIRC, the site search on An Tir Heralds searches both antirheralds.org and the Internal Letter archive on Badger's server. (There is also a search that can be done that only searches the ILs, but I am pretty sure I set up the main search to do both servers.)

I would go check on it to refresh my memory on how it's set up, but I have to leave the house right now.