Skip to comments.
What is crawling/spidering?
Free Republic ^
| Tuesday, November 2, 2004
| Momaw Nadon
Posted on 11/02/2004 11:52:22 AM PST by Momaw Nadon
What is crawling/spidering?
TOPICS: Computers/Internet; Focus Software; Free Republic Policy/Q&A; Reference
KEYWORDS: crawling; spidering
Navigation: use the links below to view more comments.
first 1-20, 21-39 next last
Can anyone explain what crawling/spidering is?
To: Momaw Nadon
Just wondering the same thing....what's the Geek Gibberish?
2
posted on
11/02/2004 11:53:00 AM PST
by
americanMel
(W...The President)
To: Momaw Nadon
I was wondering that myself...
3
posted on
11/02/2004 11:53:12 AM PST
by
WinOne4TheGipper
(Click my profile page to see my US Election Atlas Presidential Prediction Map!)
To: Momaw Nadon
It's a WWW bot thing. Has nothing to do with you, just disregard.
4
posted on
11/02/2004 11:53:14 AM PST
by
TBarnett34
(Can I get an UNNNGH?!)
To: Momaw Nadon
Yeah... if you all are gonna tell us not to do something... can you at least explain what it is you dont want us to do? Crawling/spidering?? what kind of jargon is that?
5
posted on
11/02/2004 11:53:15 AM PST
by
Betaille
(Harry Potter is a Right-Winger)
To: Momaw Nadon
I think that means trolling!
6
posted on
11/02/2004 11:53:21 AM PST
by
Halls
To: Momaw Nadon
This has made me paranoid. I thought it might just be me. Like maybe I hit reload to many times. Creepy. I sent a private request for info on it.
7
posted on
11/02/2004 11:54:19 AM PST
by
Revel
To: Momaw Nadon
Don't use a 'bot' program to filter through the content on the website. Okay?
8
posted on
11/02/2004 11:54:20 AM PST
by
atomicpossum
(If there are two Americas, John Edwards isn't qualified to lead either of them.)
To: Momaw Nadon
9
posted on
11/02/2004 11:54:45 AM PST
by
KidGlock
(I already voted for Bush/Cheney 2004. Did you?)
To: Halls
no, it doesn't.
robots.txt puts limits on what pages the web search engine 'spiders' can read and catalog. No robots.txt or spiders ignoring it = all pages spidered, big use of resources.
10
posted on
11/02/2004 11:55:02 AM PST
by
flashbunny
(Every thought that enters my head requires its own vanity thread.)
To: Momaw Nadon
Thank you for asking. I've no clue what it means. Sounds creepy though, doesn't it? Spidering...icky.
To: Momaw Nadon
Methinks FR is sending a message to some pernicious hackers...
12
posted on
11/02/2004 11:55:06 AM PST
by
JennysCool
(Terrorism: Not a global test, John, but a pop quiz.)
To: Momaw Nadon
ENOUGH WITH THE VANITIES!!!
13
posted on
11/02/2004 11:55:44 AM PST
by
ServesURight
(Tim Michels for U.S. Senate Wisconsin)
To: atomicpossum
I guess I don't have to worry, since I have NO CLUE what a "bot" program is.
14
posted on
11/02/2004 11:55:45 AM PST
by
Not A Snowbird
(Official RKBA Landscaper and Arborist, Pajama Duchess of Green Leafy Things)
To: Momaw Nadon
Folks like google, using computer programs to read and index web pages.
Every major site has a robots.txt page that tells spyders which pages to index and which ones to skip.
Robots can also put a drag on a site by requesting lots of pages. It slows the site down for everyone. A malicious robot can deliberately attempt to slow a site down. Thi is called a Denial of Service (DOS) attack.
15
posted on
11/02/2004 11:56:22 AM PST
by
js1138
(D*mn, I Missed!)
To: Momaw Nadon
It's a Peter Parker thing......
16
posted on
11/02/2004 11:56:23 AM PST
by
tacticalogic
("Oh bother!" said Pooh, as he chambered his last round.)
To: Momaw Nadon
Does this have anything to do with a water spout? I am not a computer geek and don't speak the language.
To: Momaw Nadon
It's an automatic program (called a robot) that follows all the links and records all the content it comes across (a process called indexing). It's how search engines like google finds the content that you search. The Robots.txt file tells robots what's permissible to access. All the indexing takes bandwidth from the site, so some sites are very restrictive on how much a robot is allowed to download.
18
posted on
11/02/2004 11:56:53 AM PST
by
John Jorsett
(Kerry-Edwards: FORGING AHEAD)
To: Momaw Nadon
Search engines "crawl" or "spider" the internet looking for sites and pages to add to their database.
19
posted on
11/02/2004 11:56:57 AM PST
by
Bob J
(Rightalk.com...coming soon!)
To: Momaw Nadon
It's what Kerry and the democrats are going to be doing after they lose the election.
Actually, I think it's just some remote program that is trying to access FR and the mods are telling whoever is doing it to knock it off.
Navigation: use the links below to view more comments.
first 1-20, 21-39 next last
Disclaimer:
Opinions posted on Free Republic are those of the individual
posters and do not necessarily represent the opinion of Free Republic or its
management. All materials posted herein are protected by copyright law and the
exemption for fair use of copyrighted works.
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson