Free Republic
Browse · Search
General/Chat
Topics · Post Article

Skip to comments.

10 Search Engines to Explore the Invisible Web
MakeUseOf.com ^ | Mar. 14th, 2010 | Saikat Basu

Posted on 03/23/2010 3:15:34 AM PDT by Daffynition

No, it’s not Spiderman’s latest web slinging tool but something that’s more real world. Like the World Wide Web.

The Invisible Web refers to the part of the WWW that’s not indexed by the search engines. Most of us think that that search powerhouses like Google and Bing are like the Great Oracle…they see everything. Unfortunately, they can’t because they aren’t divine at all; they are just web spiders who index pages by following one hyperlink after the other.

But there are some places where a spider cannot enter. Take library databases which need a password for access. Or even pages that belong to private networks of organizations. Dynamically generated web pages in response to a query are often left un-indexed by search engine spiders.


Search engine technology has progressed by leaps and bounds. Today, we have real time search and the capability to index Flash based and PDF content. Even then, there remain large swathes of the web which a general search engine cannot penetrate. The term, Deep Net, Deep Web or Invisible Web lingers on.

To get a more precise idea of the nature of this ‘Dark Continent’ involving the invisible and web search engines, read what Wikipedia has to say about the Deep Web. The figures are attention grabbers – the size of the open web is 167 terabytes. The Invisible Web is estimated at 91,000 terabytes. Check this out – the Library of Congress, in 1997, was figured to have close to 3,000 terabytes!

How do we get to this mother load of information?

That’s what this post is all about. Let’s get to know a few resources which will be our deep diving vessel for the Invisible Web. Some of these are invisible web search engines with specifically indexed information.

Infomine

invisible web search engines

Infomine has been built by a pool of libraries in the United States. Some of them are University of California, Wake Forest University, California State University, and the University of Detroit. Infomine ‘mines’ information from databases, electronic journals, electronic books, bulletin boards, mailing lists, online library card catalogs, articles, directories of researchers, and many other resources.

You can search by subject category and further tweak your search using the search options. Infomine is not only a standalone search engine for the Deep Web but also a staging point for a lot of other reference information. Check out its Other Search Tools and General Reference links at the bottom.

The WWW Virtual Library

invisible web search engines

This is considered to be the oldest catalog on the web and was started by started by Tim Berners-Lee, the creator of the web. So, isn’t it strange that it finds a place in the list of Invisible Web resources? Maybe, but the WWW Virtual Library lists quite a lot of relevant resources on quite a lot of subjects. You can go vertically into the categories or use the search bar. The screenshot shows the alphabetical arrangement of subjects covered at the site.

Intute

invisible web search engines

Intute is UK centric, but it has some of the most esteemed universities of the region providing the resources for study and research. You can browse by subject or do a keyword search for academic topics like agriculture to veterinary medicine. The online service has subject specialists who review and index other websites that cater to the topics for study and research.

Intute also provides free of cost over 60 free online tutorials to learn effective internet research skills. Tutorials are step by step guides and are arranged around specific subjects.

Complete Planet

search invisible web

Complete Planet calls itself the ‘front door to the Deep Web’. This free and well designed directory resource makes it easy to access the mass of dynamic databases that are cloaked from a general purpose search. The databases indexed by Complete Planet number around 70,000 and range from Agriculture to Weather. Also thrown in are databases like Food & Drink and Military.

For a really effective Deep Web search, try out the Advanced Search options where among other things, you can set a date range.

Infoplease

search invisible web

Infoplease is an information portal with a host of features. Using the site, you can tap into a good number of encyclopedias, almanacs, an atlas, and biographies. Infoplease also has a few nice offshoots like Factmonster.com for kids and Biosearch, a search engine just for biographies.

DeepPeep

search invisible web

DeepPeep aims to enter the Invisible Web through forms that query databases and web services for information. Typed queries open up dynamic but short lived results which cannot be indexed by normal search engines. By indexing databases, DeepPeep hopes to track 45,000 forms across 7 domains.

The domains covered by DeepPeep (Beta) are Auto, Airfare, Biology, Book, Hotel, Job, and Rental. Being a beta service, there are occasional glitches as some results don’t load in the browser.

IncyWincy

how to use the invisible web

IncyWincy is an Invisible Web search engine and it behaves as a meta-search engine by tapping into other search engines and filtering the results. It searches the web, directory, forms, and images. With a free registration, you can track search results with alerts.

DeepWebTech

how to use the invisible web

DeepWebTech gives you five search engines (and browser plugins) for specific topics. The search engines cover science, medicine, and business. Using these topic specific search engines, you can query the underlying databases in the Deep Web.

Scirus

how to use the invisible web

Scirus has a pure scientific focus. It is a far reaching research engine that can scour journals, scientists’ homepages, courseware, pre-print server material, patents and institutional intranets.

TechXtra

TechXtra concentrates on engineering, mathematics and computing. It gives you industry news, job announcements, technical reports, technical data, full text eprints, teaching and learning resources along with articles and relevant website information.

Just like general web search, searching the Invisible Web is also about looking for the needle in the haystack. Only here, the haystack is much bigger. The Invisible Web is definitely not for the casual searcher. It is a deep but not dark because if you know what you are searching for, enlightenment is a few keywords away.

Do you venture into the Invisible Web? Which is your preferred search tool?

Image credit: MarcelGermain


TOPICS: Computers/Internet; Education; History
KEYWORDS: deepnet; deepweb; internet; invisibleweb; pl; searchengine; searchengines
Navigation: use the links below to view more comments.
first 1-5051-62 next last
FYI
1 posted on 03/23/2010 3:15:34 AM PDT by Daffynition
[ Post Reply | Private Reply | View Replies]

To: Daffynition

very interesting. Gonna have to investigate.


2 posted on 03/23/2010 3:18:59 AM PDT by RC one (WHAT!!!!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: RC one

There are a couple listed I have not even heard of before.

Just what we need ...another excuse to spend more time on the ‘net! BWAHAHAHA!


3 posted on 03/23/2010 3:23:08 AM PDT by Daffynition (What's all this about hellfire and Dalmatians?)
[ Post Reply | Private Reply | To 2 | View Replies]

To: Daffynition

ping for later


4 posted on 03/23/2010 3:27:40 AM PDT by Upstate NY Guy (Gen 15:16 The iniquity of the Amorite is not yet complete.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

cool - I have been frustrated lately finding the same old links on every big-name search engine


5 posted on 03/23/2010 3:30:04 AM PDT by Puddleglum ("due to the record harvest, rationing will continue as usual")
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Bump


6 posted on 03/23/2010 3:32:41 AM PDT by Skooz (Gabba Gabba we accept you we accept you one of us Gabba Gabba we accept you we accept you one of us)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

bump


7 posted on 03/23/2010 3:33:06 AM PDT by Tuscaloosa Goldfinch ( T.G., global warming denier.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Great post. I wanted alternatives to Obamatron’s Google.


8 posted on 03/23/2010 3:41:08 AM PDT by SueRae
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

pagemarker....thank you!


9 posted on 03/23/2010 3:44:17 AM PDT by DrewsMum (Am I the only one that forgets I'm not on FB & tries to "like" freeper comments?)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Bookmarked ‘em for investigation.


10 posted on 03/23/2010 3:44:26 AM PDT by crazyhorse691 (Now that the libs are in power dissent is not only unpatriotic, but, it is also racist.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

bookmark


11 posted on 03/23/2010 3:45:54 AM PDT by muir_redwoods (Obama: Chauncey Gardiner without the homburg)
[ Post Reply | Private Reply | To 1 | View Replies]

To: theKid51; BabyBMW

ping


12 posted on 03/23/2010 3:49:46 AM PDT by bmwcyle (Free the Navy Seals)
[ Post Reply | Private Reply | To 1 | View Replies]

bump


13 posted on 03/23/2010 3:50:28 AM PDT by Non-Sequitur
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Thanks for the post.


14 posted on 03/23/2010 3:52:18 AM PDT by Former Proud Canadian (How do I change my screen name now that we have the most conservative government in the world?)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Thanks!


15 posted on 03/23/2010 3:53:04 AM PDT by Adder (Proudly ignoring Zero since 1-20-09! WTFU!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

bump


16 posted on 03/23/2010 3:56:07 AM PDT by dangerdoc
[ Post Reply | Private Reply | To 1 | View Replies]

bttt


17 posted on 03/23/2010 3:56:37 AM PDT by Matchett-PI (Sowell's book, Intellectuals and Society, eviscerates the fantasies that uphold leftist thought)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Thanks. Bookmark.


18 posted on 03/23/2010 4:19:07 AM PDT by Issaquahking (Help Sarah Palin! go to - http://www.conservatives4palin.com - You know what to do!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Thank you!


19 posted on 03/23/2010 4:26:47 AM PDT by wally_bert (It's sheer elegance in its simplicity! - The Middleman)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

I’m sure someday I’ll get to the end of the last FreeRepublic thread and then start exploring the rest of the internet.

So, bookmark for potential future use.


20 posted on 03/23/2010 4:33:02 AM PDT by Rocky (REPEAL IT!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

bfl


21 posted on 03/23/2010 4:38:44 AM PDT by mplsconservative (0bama = Epic Fail)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition
bttt
22 posted on 03/23/2010 4:41:50 AM PDT by BA63
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

THX


23 posted on 03/23/2010 4:52:52 AM PDT by gigster
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Thank you very much for the post. BTTT.


24 posted on 03/23/2010 5:12:35 AM PDT by PA Engineer (Liberate America from the occupation media.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

bookmark


25 posted on 03/23/2010 5:22:08 AM PDT by wvguy (Montani semper liberi)
[ Post Reply | Private Reply | To 1 | View Replies]

To: LucyT

Worth a ping to the list? More sources to explore...


26 posted on 03/23/2010 5:23:44 AM PDT by Protect the Bill of Rights ( COWARD, Thy name is STUPAK.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

ping


27 posted on 03/23/2010 5:27:39 AM PDT by rjsimmon (1-20-2013 The Tree of Liberty Thirsts)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

B4L8r


28 posted on 03/23/2010 5:56:47 AM PDT by AFreeBird
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition
The public web is only a few hundred terabytes?

To put things in perspective, I have about 6 terabytes of storage in my house.

29 posted on 03/23/2010 6:18:11 AM PDT by dangerdoc
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Bookmarking,

Thanks, this is very cool.


30 posted on 03/23/2010 6:51:57 AM PDT by King_Corey (www.kingcorey.com)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition; rdb3; Calvinist_Dark_Lord; GodGunsandGuts; CyberCowboy777; Salo; Bobsat; JosephW; ...

31 posted on 03/23/2010 6:53:41 AM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Thank you for posting this, Daffynition. I forwarded it to my daughters. They might be able to make use of it in what they for work. They’ve been able to use info supplied by FReepers on a couple of occasions.


32 posted on 03/23/2010 7:03:13 AM PDT by My hearts in London - Everett (So the writer who breeds more words than he needs, is making a chore for the reader who reads.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Thanks, very helpful.


33 posted on 03/23/2010 7:04:19 AM PDT by buschbaby (Resist! Repeal! Rejoice!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

PING


34 posted on 03/23/2010 7:05:38 AM PDT by Logic n' Reason (We are all nine meals short of total anarchy.....think about it.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

There are still gopher, archie, and veronica servers out there. (Old timers will know what I mean)


35 posted on 03/23/2010 7:52:25 AM PDT by zeugma (Proofread a page a day: http://www.pgdp.net/)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition
> ...the Library of Congress, in 1997, was figured to have close to 3,000 terabytes!

Oh, really???

The source that (mistaken) claim (http://en.wikipedia.org/wiki/Deep_Web#cite_note-2) is based on disagrees:

http://www.lesk.com/mlesk/ksg97/ksg.html

That says it was 20 TB, not 3000 TB.

Or am I reading this wrong?


36 posted on 03/23/2010 8:29:31 AM PDT by dayglored (Listen, strange women lying in ponds distributing swords is no basis for a system of government!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: dayglored; Daffynition
Oops, nevermind, my bad.

That chart didn't include all the audio and video recording files. When you add that, it gets to 3000 TB.

Yeah, that was probably about right.

37 posted on 03/23/2010 8:34:15 AM PDT by dayglored (Listen, strange women lying in ponds distributing swords is no basis for a system of government!)
[ Post Reply | Private Reply | To 36 | View Replies]

To: dayglored

If it were only 20 TB, it would be theoretically posible to mirror the entire internet in your basement with a few thousand dollars worth of hard drives.

I thought that the internet was being measured in petabytes now.


38 posted on 03/23/2010 8:36:06 AM PDT by dangerdoc
[ Post Reply | Private Reply | To 36 | View Replies]

To: dayglored

“Estimating that is a fairly difficult task, but one person made an estimate not so long ago who can probably be trusted to have a good idea. Eric Schmidt, the CEO of Google, the world’s largest index of the Internet, estimated the size at roughly 5 million terabytes of data. That’s over 5 billion gigabytes of data, or 5 trillion megabytes. Schmidt further noted that in its seven years of operations, Google has indexed roughly 200 terabytes of that, or .004% of the total size.”

http://www.wisegeek.com/how-big-is-the-internet.htm


39 posted on 03/23/2010 8:41:31 AM PDT by dangerdoc
[ Post Reply | Private Reply | To 36 | View Replies]

To: Daffynition

Great list, thanks.

Here’s another that will get you going: http://www.wolframalpha.com/

It not only digs out the info, but can figure out the right answer. Go hit the link for Stephen Wolfram’s Info, it’s a 13 minute video that shows the power W|A has under the covers.


40 posted on 03/23/2010 9:23:06 AM PDT by brityank (The more I learn about the Constitution, the more I realise this Government is UNconstitutional !! )
[ Post Reply | Private Reply | To 1 | View Replies]

To: dangerdoc
Remember, that figure (20TB) was from 1997, 13 years ago.

The web's content has grown by literally orders of magnitude since then.

BTW, being something of a proper-English nut, I hate it when people use the terms "literally" and "order of magnitude" improperly. But the above is indeed true (at least 10^2), so I'll take the grammar risk.

If the google quote is right, 5,000,000,000 TB, then even my statement falls short by many orders of magnitude.

41 posted on 03/23/2010 10:49:36 AM PDT by dayglored (Listen, strange women lying in ponds distributing swords is no basis for a system of government!)
[ Post Reply | Private Reply | To 39 | View Replies]

To: dayglored; dangerdoc
oh crap, I can't cut and paste any more.

> If the google quote is right, 5,000,000,000 TB

should have been

If the google quote is right, 5,000,000 TB

42 posted on 03/23/2010 10:52:03 AM PDT by dayglored (Listen, strange women lying in ponds distributing swords is no basis for a system of government!)
[ Post Reply | Private Reply | To 41 | View Replies]

To: Daffynition

Maybe I missed something, but how do these search engines access private networks?


43 posted on 03/23/2010 10:54:46 AM PDT by Straight Vermonter (Congratulations UVM Catamounts - Going to the Big Dance!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: dayglored

“If the google quote is right, 5,000,000 TB”

How much of that is porn, pirated movies and left wing blather? Makes you wonder if this whole interweb thing was a good idea.


44 posted on 03/23/2010 11:27:41 AM PDT by dangerdoc
[ Post Reply | Private Reply | To 42 | View Replies]

To: Daffynition

For later.


45 posted on 03/23/2010 11:29:28 AM PDT by Lurker (The avalanche has begun. The pebbles no longer have a vote.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition
The deep Web is several orders of magnitude larger than the surface Web.[2]

from Wekipedia....

46 posted on 03/23/2010 1:27:46 PM PDT by GOPJ (http://hisz.rsoe.hu/alertmap/index2.php?area=dam&lang=eng)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Great post - thanks.


47 posted on 03/23/2010 1:38:05 PM PDT by GOPJ (http://hisz.rsoe.hu/alertmap/index2.php?area=dam&lang=eng)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Puddleglum

It’s going to be fun exploring some of these links....or an enormous waste of time!


48 posted on 03/23/2010 3:39:31 PM PDT by Daffynition (What's all this about hellfire and Dalmatians?)
[ Post Reply | Private Reply | To 5 | View Replies]

To: Daffynition

Interesting, but the phrase is ‘mother lode’, not ‘load’.

/Pedantic mode


49 posted on 03/23/2010 3:50:48 PM PDT by Riley (The Fourth Estate is the Fifth Column.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Daffynition

Bookmarked...


50 posted on 03/23/2010 6:26:49 PM PDT by BlessedBeGod (New Wizard of Oz: Pelosi as the Wicked Witch of the West & Michelle as the Wicked Witch of the East.)
[ Post Reply | Private Reply | To 1 | View Replies]


Navigation: use the links below to view more comments.
first 1-5051-62 next last

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
General/Chat
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson