Posted on 05/04/2006 12:14:59 PM PDT by nickcarraway
Webmasters have been seething at Google since it introduced its 'Big Daddy' update in January, the biggest revision to the way its search engine operates for years.
Alarm usually accompanies changes to Google's algorithms, as the new rankings can cause websites to be demoted, or disappear entirely. But four months on from the introduction of "Big Daddy," it's clear that the problem is more serious than any previous revision - and it's getting worse.
Webmasters now report sites not being crawled for weeks, with Google SERPS (search engine results pages) returning old pages, and failing to return results for phrases that used to bear fruitful results.
"Some sites have lost 99 per cent of their indexed pages," reports one member of the Webmaster World forum. "Many cache dates go back to 2004 January." Others report long-extinct pages showing up as "Supplemental Results."
This thread (http://groups.google.com/group/alt.internet.search-engines/browse_thread/thread/5a15acac5a0245ce/b429366a28a29507#b429366a28a29507) is typical of the problems.
With creating junk web pages is so cheap and easy to do, Google is engaged in an arms race with search engine optimizers. Each innovation designed to bring clarity to the web, such as tagging, is rapidly exploited by spammers or site owners wishing to harvest some classified advertising revenue.
Recently, we featured a software tool that can create 100 Blogger weblogs in 24 minutes, called Blog Mass Installer. A subterranean industry of sites providing "private label articles," or PLAs exists to flesh out "content" for these freshly minted sites. And as a result, legitimate sites are often caught in the cross fire.
But the new algorithms may not be solely to blame. Google's chief executive Eric Schmidt has hinted at another reason for the recent chaos. In Google's earnings conference call last month, Schmidt was frank about the extent of the problem.
"Those machines are full," he said (http://www.iht.com/articles/2006/04/21/business/GOOGLE.php). "We have a huge machine crisis."
And there's at least some anecdotal evidence to support the theory that hardware limitations are to blame.
"The issue I have now is Googlebot is SLAMMING my sites since last week, but none of it makes it into the index. If it's old pages being re-indexed or new pages for the first page, they don't show up," writes one webmaster.
The confusion has several consequences which we've rarely seen discussed outside web circles.
Giving Google the benefit of the doubt, and assuming the changes are intentional, one webmaster writes: "In which case Google's index, and hence effectively 'the Web as most people know it' is set to become a whole lot smaller in the coming weeks."
It's barely more than a year since Yahoo! and Google were engaged in a willy-waving exercise to claim who had the largest index. (See My spam-filled search index is bigger than yours! (http://www.theregister.co.uk/2005/08/16/google_yahoo_junk/))
Now size, it seems, doesn't matter.
There's also the intriguing question raised by search engines that are unable to distinguished between nefarious sites and legitimate SEO (search engine optimization) techniques? The search engines can't, we now know, blacklist a range of well-establish techniques without causing chaos. In future, will the search engines need to code for backward bug compatibility?
And lingering in the background is the question of whether the explosion of junk content - estimates put robot-generated spam consists of anywhere between one-fifth and one-third of the Google index - can be tamed?
"At this rate," writes one poster on the Google Sitemaps Usenet group, in a year the SERPS will be nothing but Amazon affiliates, Ebay auctions, and Wiki clones. Those sites don't seem to be affected one bit by supplemental hell, 301s, and now deindexing."
With $8 billion in the bank, Google is better resourced and more focussed than anyone - but it's still struggling. Financial analysts noted that its R&D expenditure now matches that of a wireline telco.
Only a cynic would suggest that poor SERPs drive desperate businesses to the search engines own classified ad departments - so if you want to play, you have to pay. Banish that unworthy thought at once.
ping
I love Google, I don't care how liberal they are.
I have to say I find google less efficient these days (privacy issues?), but they are still more on the ball than any of the others. Pretty much tried them all. If anyone has good suggestions, clue me.
Try www.DogPile.com
I am not associated with anyone!
The best one ever was Northern Lights, but it went to paid only several years ago. I got much more on that one than I ever did on Google, although that was the second best. How I have missed it!
I've used:
dogpile, yahoo, lycos, webcrawler, ixmetasearch, clusty, and dmoz
and I think these are among the better ones, but still, a flawed google produces more good results
I don't think Google, "IMPROVED." I think they, sold out, to the spammers and hackers. The end result, is that they created more problems than they supposedly, solved...
Yes, that was a good one. Wasn't it taken over by overture (which also isn't bad, but not among the very best). I like Highbeem for news headlines. Very nice. Technorati is good for blogs.
I agree, they sold out. I used an engine called Wisenut, which is still around, but after it was swallowed up by Looksmart it seemed like they "dumbed it down". I feel the same way about Google. As I posted before, not as good, but still better than the rest.
I really don't know what happend to my Northern Lights. I just know that I've missed it a lot. I do a lot of genealogical research online and it yielded a lot more dead people that I needed that any other. It was excellent on current news too.
Some folks obsess over search engine rankings.
I quit using it because of FR reports as to Google’s leftist tendencies. Plenty of alternatives out there.
Far and away the most important part of the article.
I used to use google the way I would use the encyclopedia, now it's more like the yellow pages.
If that's your thing, I like to bypass ancestry.com and use rootsweb - you can free search the SSDI. The Mormon's have an amazing site called familysearch.org - traced my husband's family back to the early 1800s and all of my Jewish relatives in Europe! No idea how they gleaned all this stuff. Great international links.
Got any good URL's for alternate search engines?
Yes, I love familysearch. I've found great stuff there. I, too, avoid ancestry. A lot of my research is now in storage and I'm not going to shell out money until I have everything in front of me. How exciting about the info you've found on your family. My major family is from the burned counties of Virginia, so research is difficult.
That's probably why they did it. They were so far ahead, they figured it'd be okay to, "get a little stupid" for $$$$...
ask.com got a good review in the WSJ. I use Google but mostly out of force of habit.
Very cool, thanks for the advice - anyone else have any genealogical advice? I've been thinking about getting back into it and tracing down both my family tree and my husband's family tree. I used to have some pretty good charts, but they've gotten lost over time...
I've tried one called brainboost.com that seems interesting.
What's especially strange about that is that their cached version of a page tends to be pretty up to date. So why aren't their robots reporting when they revisit a site and it's kaput?
I've largely given up on Google. I use Yahoo and A9 (which no longer gets its results from Google).
And when I do have to use Google, I do it with Firefox, with the proper extensions installed to insure I don't see their ads and they don't get to track my actions.
I still do over half of my searching with Google, but it's getting less satisfactory all the time. I'm seldom looking for porn or something to buy.
Nor has Google done much with expanding beyond regular old listings and using things such as clustering such as Vivisimo which is my current favorite.
I have a great example from just a couple of days ago; I needed tech info on a product. Searched Google and got a bunch of sites selling it. Fired it up through Vivisimo and got the tech spec page right away.
I've used vivisimo, and it's not bad. If you want consumer advise, I really like epinions. They have not steered me wrong yet.
I've found A9 more than a little "hamstrung" of late. I like their add on features though.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.