robots.txt?

[PC Radio Net]

Does anyone know why I'm getting these statements?

dest: 207.46.98.116] Invalid resource request(/robots.txt)

[NJK]

quick , close all your ports , shut down your system and don't bring it back online before 4 weeks have past.



or you can look closely at what the line tells you and you see that a /robot.txt has tried to make contact with your server ( search engine or other netcrawlers) and they couldn't make contact due to invalid resource request.

your stream is only availible for players that can handle streaming media.

[djSpinnerCee]

A webcrawler (or [ro]bot), must examine robots.txt for restricted paths before crawling your site (and the links it contains) -- a DNAS does not have a robots.txt, also, if the robot does not have a mozilla useragent, the DNAS will give that invalid resource request.

It's not a bad thing... Your "hits" may explode if this bot likes what it finds.

*search engine optimizing hint: /played.html lists track titles -- people search for tracks.... how many are you listing? The max is 20.

*** Do you have a link to your website on your station page? ***

[PC Radio Net]

I'm listing 10 tracks and yes, I do have a link to my site on there. Thanks!

[djSpinnerCee]

It just means -- They're coming...

[rockouthippie]

You must have a link to your "http" shoutcast page on your web. Or someone else does.

This probably isn't anything evil. The spider in question is asking for your robots.txt file, so it's probably not an evil spider.

If it was a "bad bot" it would ignore the robots.txt file anyway. And probably not even look at it.

Sounds like a search engine found your html page shoutcast page. If that is bad, change your port and if you want the page to be hidden from search engines, put the link to your shoutcast html page in a separate directory. Like /norobots/myservers.html

In that directory (/norobots) also include a robots.txt file contains two simple words.

---cut---robots.txt
go away
---cut--

"no cache" will also prevent Google and others from archiving your pages.

If you wish to conceal links that have already been spidered, change the port of your shoutcast server.

If you go to alexa.com, you'll find some pretty good documentation on robots.txt.

[CraigF]

www.robotstxt.org