Great Circle Associates List-Managers
(February 1998)

Indexed By Date: [Previous] [Next] Indexed By Thread: [Previous] [Next]

Subject: Re: Archives and robots.txt
From: Chuq Von Rospach <chuqui @ plaidworks . com>
Date: Wed, 4 Feb 1998 21:04:50 -0800
To: cnorman @ best . com, list-managers @ GreatCircle . COM
Cc: cnorman @ shell7 . ba . best . com
In-reply-to: <199802050241 . SAA06128 @ shell7 . ba . best . com>
References: <v04003a31b0fe95333d9a @ [17 . 219 . 12 . 172]> (message from Chuq Von Rospach on Wed, 4 Feb 1998 14:00:16 -0800)

At 6:41 PM -0800 2/4/98, Cyndi Norman wrote:
> Are there global search engines I have to worry about that do FTP sites?

They don't search FTP sites per-se, but if you have urls pointing to
stuff on an FTP site, then when the search engine wanders across your
links, the data on the FTP site will be added by many of them. This
isn't a protection from the search engines, unless you put a robots.txt
on your ftp site (and do so via an HTTP daemon).

> I don't really understand this though.  If I have a custom domain that's
> really a virtual domain, why would robots.txt work?

It should work if (and only if) <> is
readable. If your virtual hosting doesn't allow for that, it won't. And
it's also possible that places like Alta vista aren't honoring the
Host: fields, and that could be causing problems....

Chuq Von Rospach (Hockey fan? <>)
Apple Mail List Gnome (mailto:chuq @
 apple .
Plaidworks Consulting (mailto:chuqui @
 plaidworks .
<> + <>

Indexed By Date Previous: Re: AOL spam controls question
From: Berg <berg @ eskimo . com>
Next: another AOL Mail Controls question
From: dattier @ wwa . com (David W. Tamkin)
Indexed By Thread Previous: Re: Archives and robots.txt
From: Cyndi Norman <cnorman @ best . com>
Next: Re: Archives and robots.txt
From: Chris Pepper <pepper-list @ list . audubon . org>

Search Internet Search