Free Republic
Browse · Search
General/Chat
Topics · Post Article

Can anyone explain what crawling/spidering is?
1 posted on 11/02/2004 11:52:22 AM PST by Momaw Nadon
[ Post Reply | Private Reply | View Replies ]


Navigation: use the links below to view more comments.
first 1-2021-27 next last
To: Momaw Nadon

Just wondering the same thing....what's the Geek Gibberish?


2 posted on 11/02/2004 11:53:00 AM PST by americanMel (W...The President)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

I was wondering that myself...


3 posted on 11/02/2004 11:53:12 AM PST by WinOne4TheGipper (Click my profile page to see my US Election Atlas Presidential Prediction Map!)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

It's a WWW bot thing. Has nothing to do with you, just disregard.


4 posted on 11/02/2004 11:53:14 AM PST by TBarnett34 (Can I get an UNNNGH?!)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

Yeah... if you all are gonna tell us not to do something... can you at least explain what it is you dont want us to do? Crawling/spidering?? what kind of jargon is that?


5 posted on 11/02/2004 11:53:15 AM PST by Betaille (Harry Potter is a Right-Winger)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

I think that means trolling!


6 posted on 11/02/2004 11:53:21 AM PST by Halls
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

This has made me paranoid. I thought it might just be me. Like maybe I hit reload to many times. Creepy. I sent a private request for info on it.


7 posted on 11/02/2004 11:54:19 AM PST by Revel
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon
Don't use a 'bot' program to filter through the content on the website. Okay?
8 posted on 11/02/2004 11:54:20 AM PST by atomicpossum (If there are two Americas, John Edwards isn't qualified to lead either of them.)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

bump for reading later


9 posted on 11/02/2004 11:54:45 AM PST by KidGlock (I already voted for Bush/Cheney 2004. Did you?)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

Thank you for asking. I've no clue what it means. Sounds creepy though, doesn't it? Spidering...icky.


11 posted on 11/02/2004 11:55:05 AM PST by Letitring
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

Methinks FR is sending a message to some pernicious hackers...


12 posted on 11/02/2004 11:55:06 AM PST by JennysCool (Terrorism: Not a global test, John, but a pop quiz.)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

ENOUGH WITH THE VANITIES!!!


13 posted on 11/02/2004 11:55:44 AM PST by ServesURight (Tim Michels for U.S. Senate Wisconsin)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

Folks like google, using computer programs to read and index web pages.

Every major site has a robots.txt page that tells spyders which pages to index and which ones to skip.

Robots can also put a drag on a site by requesting lots of pages. It slows the site down for everyone. A malicious robot can deliberately attempt to slow a site down. Thi is called a Denial of Service (DOS) attack.


15 posted on 11/02/2004 11:56:22 AM PST by js1138 (D*mn, I Missed!)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

It's a Peter Parker thing......


16 posted on 11/02/2004 11:56:23 AM PST by tacticalogic ("Oh bother!" said Pooh, as he chambered his last round.)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon
Does this have anything to do with a water spout? I am not a computer geek and don't speak the language.
17 posted on 11/02/2004 11:56:27 AM PST by stayathomemom
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

It's an automatic program (called a robot) that follows all the links and records all the content it comes across (a process called indexing). It's how search engines like google finds the content that you search. The Robots.txt file tells robots what's permissible to access. All the indexing takes bandwidth from the site, so some sites are very restrictive on how much a robot is allowed to download.


18 posted on 11/02/2004 11:56:53 AM PST by John Jorsett (Kerry-Edwards: FORGING AHEAD)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

Search engines "crawl" or "spider" the internet looking for sites and pages to add to their database.


19 posted on 11/02/2004 11:56:57 AM PST by Bob J (Rightalk.com...coming soon!)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon
It's what Kerry and the democrats are going to be doing after they lose the election.

Actually, I think it's just some remote program that is trying to access FR and the mods are telling whoever is doing it to knock it off.

20 posted on 11/02/2004 11:57:28 AM PST by rightwingreligiousfanatic (Bush/Cheney: Hope is here!)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

It means having a program walk through your web site.

Either they are indexing (like google) or data mining for info.

Robots.txt is a file made available for such programs which is supposed to tell these programs what the rules are.

Being firewalled means configuring the hardware so that the machines doing this can't get access to Jim's servers.


21 posted on 11/02/2004 11:58:08 AM PST by dinasour (Pajamahadeen)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

Another reason to restrict robots here is that the Dems apparently are using FR content in order to find attack material on conservatives. We had someone here who had some of his postings thrown back in his face in a political race (I think). I forget the details, but others can probably supply them.


23 posted on 11/02/2004 11:59:34 AM PST by John Jorsett (Kerry-Edwards: FORGING AHEAD)
[ Post Reply | Private Reply | To 1 | View Replies ]

To: Momaw Nadon

Dear god.


24 posted on 11/02/2004 11:59:36 AM PST by cwiz24 (Hey Yankees fans---Now who's ya daddy?)
[ Post Reply | Private Reply | To 1 | View Replies ]


Navigation: use the links below to view more comments.
first 1-2021-27 next last

Free Republic
Browse · Search
General/Chat
Topics · Post Article


FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson