I’ve been maintaining a website for my church, and I’ve recently been
dismayed to find the guestbook filled with spam, ads, and quite
disgusting trash (I don’t need to be specific; I’m sure you’ve seen this
stuff in your email). I had gone to quite a lot of trouble to learn perl for
the guestbook, so I wasn’t easily convinced to discard the guestbook
concept. The guestbook does garner really valid and useful comments
at times.

I first tried a few ideas like suppressing the ability for a guestbook
writer to add his website URL. These ideas didn’t work. Finally I moved
the page by copying the file from, for example, GUEST.HTM to
GUESTX.HTM. I changed all the pointers on the other pages to
GUESTX.HTM. Nevertheless, I found that the old guestbook was
still receiving spam. New and valid comments continued on the
new page.

This made me realize that the spammers were coming to my
guestbook directly, without using my hyperlinks. They were not
just finding the guestbook in the normal way. It caused me to
attempt to put myself into the mind of such spammers. I realized
that if I wanted to do such an unkind thing, I would use a search
engines and to find the guestbooks. Tada!

I have now attempted to avoid letting search engines find my
guestbook by using the features of a file entitled ROBOTS.TXT.
I include this file on my website to discourage search engines
from listing my guestbook. True visitors to my site, however, can
find the guestbook easily.

Here is what I use in ROBOTS.TXT.  Perhaps you can use the
concept to help your sites too. I actually used lower case for the
filename, but upper case in this narrative to make it stand out.

User-Agent: *
Disallow: /guest/
Disallow: /*.log