JFB - Who is online question :)

[Charlyee]

Joe,

Sorry for bugging you again. If I look at "Who's Online" from the Quick links, other than members & guests, I see multiple entries for "Yahoo! Slurp Spider" & I also see "yahooaskjeeves".

Who/What on earth are these?

Thanks Much

[dmapr]

Originally Posted by Charlyee

Joe,

Sorry for bugging you again. If I look at "Who's Online" from the Quick links, other than members & guests, I see multiple entries for "Yahoo! Slurp Spider" & I also see "yahooaskjeeves".

Who/What on earth are these?

Thanks Much

I know I'm not JFB, but having worked with a spider bot before I can give you some info. Google, Yahoo search and other services will "crawl" through the WWW, loading, storing, indexing, analyzing and what not different pages. "Good" bots follow the rules (sites can limit the bot access by using robots.txt file to set their bot access rules) and also identify themselves similarly to the way browsers identify themselves — by setting the User-Agent header in the HTTP request sent to the site. The sites can then use these to identify the active bots.

[Charlyee]

Wirelessly posted (SAMSUNG-SGH-I607/I607FG1 Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240) UP.Link/6.3.1.17.0)

Thanks dmapr (even though you are not JFB ), very interesting. They actually show up as looking at posts just like a member/guest. So they mimic human behavior for the purpose of information gathering?

So what would be an example of a "bad" bot?

Thanks Much

[dmapr]

Originally Posted by Charlyee

Wirelessly posted (SAMSUNG-SGH-I607/I607FG1 Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240) UP.Link/6.3.1.17.0)

Thanks dmapr (even though you are not JFB ), very interesting. They actually show up as looking at posts just like a member/guest. So they mimic human behavior for the purpose of information gathering?

So what would be an example of a "bad" bot?

Thanks Much

The bots typically parse HTML looking for links and then follow them and repeat — a lot more methodically than a human would.

A "bad" bot would be a bot that shows up on the site and ignoring any directives that may've been specified in the robots.txt file proceeds to read every link as fast as they could, clogging up the server resources. Here's a Web robots FAQ, hopefully it can answer some questions — I'm about seven years removed from robots and my memory may be failing me.

[JFB]

Yup, dmapr explained it nicely. It is good to see those because it means the search engines are indexing all of the great info. we have here.

I could exclude them from the "who's online" list, but I think it is interesting to see when they are active.

[Fire14]

Wow, that is pretty wild & I never knew about them. I agree Joe leave them be if they are not hurting anything.

[SteveW]

Originally Posted by Charlyee

They actually show up as looking at posts just like a member/guest. So they mimic human behavior for the purpose of information gathering?

Originally Posted by dmapr

The bots typically parse HTML looking for links and then follow them and repeat — a lot more methodically than a human would.
...
— I'm about seven years removed from robots and my memory may be failing me.

Sounds like your memory is fine. I haven't worked with bots per se, but in the early days of the Web I did a bit of playing around to make my sites appear higher in search engines, and I also tracked bot behavior on the Web server I ran. As you implied, allowing bots is the price we pay to have the Web indexed efficiently, enabling Google and other search engines.

To Charlyee's point, they have to do essentially what a person would do, because there is only one protocol, HTTP, that all Web servers use, and the HTML pages served by that protocol have a fairly small number of commands that are relevant. Links and images are still a large part of what's out there. Tim Berners-Lee's insight was to understand that this was enough to do lots of cool stuff.

Of course, there are plenty of ways to protect your site from bots in addition robots.txt, which is essentially the honor system. You can require a login for example, like corporate intranets do. But if you want your information available to the world, you have to accommodate bots.


SW

[Charlyee]

Wirelessly posted (SAMSUNG-SGH-I607/I607FG1 Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240) UP.Link/6.3.1.17.0)

Originally Posted by JFB

Yup, dmapr explained it nicely. It is good to see those because it means the search engines are indexing all of the great info. we have here.

I could exclude them from the "who's online" list, but I think it is interesting to see when they are active.

Joe thanks, yes it is interesting to see when they are active. Now that I know what they are I will be looking for them

Steve, thanks for the additional piece of information, it made it more interesting yet.

dmapr, thanks for the link. Yes your memory is just fine, being modest aren't you.?

Fire, aren't you glad I asked, now we both learnt something additional today

[Fire14]

Originally Posted by Charlyee

Wirelessly posted (SAMSUNG-SGH-I607/I607FG1 Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240) UP.Link/6.3.1.17.0)



Joe thanks, yes it is interesting to see when they are active. Now that I know what they are I will be looking for them

Steve, thanks for the additional piece of information, it made it more interesting yet.

[v]dmapr[/b], thanks for the link. Yes your memory is just fine, being modest aren't you.?

Fire, aren't you glad I asked, now we both learnt something additional today

Yes I am glad, but I think my heads going to explode now.

[dmapr]

Originally Posted by SteveW

Sounds like your memory is fine.

Of course, there are plenty of ways to protect your site from bots in addition robots.txt, which is essentially the honor system. You can require a login for example, like corporate intranets do. But if you want your information available to the world, you have to accommodate bots.


SW

Originally Posted by Charlyee

Wirelessly posted (SAMSUNG-SGH-I607/I607FG1 Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240) UP.Link/6.3.1.17.0)

Steve, thanks for the additional piece of information, it made it more interesting yet.

dmapr, thanks for the link. Yes your memory is just fine, being modest aren't you.?

I worked on the other side of bot — writing one, so we were very particular about honoring the system. When I say my memory is failing it means I won't be able to write an honorable one without some serious refreshing

Steve's point just triggered another piece of memory — bot traps. Bot traps are usually some obscure links that lead the bots on wild goose chases, that can be hidden/disguised from the user who is browsing. For instance, you can place a number of "legitimate" looking links on your web page and hide them from view using CSS or some other attributes. Usually bots do not try to analyze the full HTML (it'll slow them down too much) to see whether the links will be visible or not and may fall into the trap.