New Numbers for the Curious (or FR Traffic Stats)

Free Republic Website Usage ^

[John Robinson]

For those that love numbers, I have three links to Free Republic usage statistics:

• Free Republic Website Usage NEW!
  compiled from server logs; ready by 2am PT daily

• Free Republic Daily Stats
  compiled from posting logs; ready by 2am PT daily

• Traffic Rank [Alexa]
  see their website

Caveat:

1. I recently upgraded the program (AWStats) used to generate the Website Usage pages. The old pages, though built with the new program, utilized the old format database which did not provide some fields. Those fields will be 0 or not present on the older reports.

2. For some reason the reports switched to GMT some months ago, I switched back to PT this month. Each switch fouled the "Hours" graph for the month of the switch.

3. We lost our senses Mar 12-14, or rather, the stat program forgot how to tell the difference between a Hit (any request to the web server) and a Page (a request for a HTML page, or similar page of content); all hits were erroneously counted as pages, the actual number of pages for that time period is actually quite a bit less.

4. And, on Wednesday Mar 12th, I enabled HTTP compression, practically halving our bandwidth usage! Just in time for the overwhelming load that day!

5. The Daily Stats page is clearly showing signs of age. It no longer collects the "Front-Side" and "Back-Side" statistics. However, you will find "Front-Side" had become a duplicate effort, see the Website Usage pages to continue on.

6. Numbers on the Daily Stats page and the new Website Usage pages are generally an even comparison. However, consider the Daily Stats pages are generated on daily GMT time boundaries, and the Website Usage pages are generated at PT (or GMT--oops) boundaries.

And now the ramble.

One thing I don't yet have for readers is a report of the actual Mbps that flows through our Internet connection. Leading up to the war we've routinely saturated our link, that's 10 Mbps, or 10 million 1's and 0's per second. Those numbers are very interesting as well!

As we reach our bandwidth limit, the network link becomes saturated and we begin to slow down. Last Wed (Mar 12) was the breaking point, the server was falling over as network congestion backed up through the kernel and into my application-- what a mess that will make!

Foreseeing impending trouble days before, I had previously installed a little gadget called mod_gzip. The mod_gzip is an Apache webserver extension that compresses HTML pages on the fly. After an abbreviated test run of just a few days I enabled mod_gzip to bail me out of this crisis.

My savior! The compression is highly effective, reducing pages by an average of 79% the original page size. However, the true savings are closer to 50%. Whether the module is used at all is dependent upon the browser used to view the page. Due to brain damage, some browsers do not work properly or at all when viewing a compressed page, and therefore must not be sent compressed pages. And images are not compressed at all-- mostly because images are already highly compressed, but partly because of even more brain damage in even more browsers.

50% isn't bad, not bad at all. We've now effectively doubled our bandwidth capacity. Have we tamed the traffic? Mostly. Bandwidth usage is now routinely in the 5-6 Mbps range, which is effectively 10-12 Mbps. Thursday morning from 7am-1pm PT we were in the 7-8 Mbps range (14-16 Mbps!) and often times pushing the ceiling, pushing our full link capacity of 10 Mbits hard for extended periods of time. That's 20 Mbits of bandwidth, 12 T-1 lines, or nearly one half T-3. Simply amazing!

[Bahbah]

I don't understand any of that, but I know I am glad you do.

[John Robinson]

I'll proofread that later. I'm going to bed.

[LowOiL]

on Wednesday Mar 12th, I enabled HTTP compression, practically halving our bandwidth usage! Just in time for the overwhelming load that day!

Wow, and we did not have to download a WinZip file to do this?

[grania]

There used to be information about how many people were logged in to FR at any given time. It was kind of a neat thing to know.

[FreedomPoster]

Thanks for all you do. Love seeing the techie minutia every now and then.

Doesn't a T-3 get cheaper than all the separate T-1's at some point, less than a T-3's worth? I know it does for voice, but I guess it might not make as big of a difference for an Internet connection.

[Admin Moderator]

From the posting log stats:

Distinct Posters on Thursday: 4,358...a new record topping the previous high of 4,313 from the day before.

Total articles + replies on Thursday: 35,055...a new record topping the previous high of 31,153 set on Tuesday.

Articles posted on Thursday: 1,222...not a new record, second most ever to the 1,301 posted the day after the 2000 elections.

Replies posted on Thursday: 33,833...a new record, topping the previous high of 30,102 on Tuesday.

New signups on Thursday: 350...the fourth highest total behind 3 days in the timeframe just after the 2000 elections.

Busy busy busy

[Free the USA]

fyi

[tictoc]

On-the-fly compression works only if it also is enabled in the "client browser" (the browser software that people use to log on to FR).

I believe that in Internet Explorer 6, this feature is already turned on by default.

Here is how to check:

Go to Start+Settings+Control Panel, then double click the Icon that says "Internet Options." Now click the tab that says "Advanced" scroll down until you see "Use HTTP 1.1"; make sure there is a check in the box and press ok. Press ok again to close Internet Options, and close any open IE windows. When you launch IE again you should see that surfing is much faster than it was before.

In addition, if you are using a proxy like Proxomitron (Proxo), WebWasher, etc., you should also turn on "Use HTTP 1.1 through proxy connections" by putting a checkmark into the box next to it.

[Quix]

FANTASMAGORICALLY WONDERFUL!

GREAT WORK JOHN. THANKS TONS!

GOD'S BEST TO THE PRES AND THE TROOPS AND YOU, THE FAMILY, AND OF COURSE, FR!

[Peach]

A fantastic job! What a resource. Hard to believe there will ever be a Democrat elected to any high office with the info Freepers are able to gather - heh heh.

[nicmarlo]

Free Republic Website Usage NEW!

You are not authorized to view this page

So, what did I do wrong JR?? : )

[Alas Babylon!]

How's your cpu load? Usually enabling heavy compression will further tax a cpu. Anything over a consistant 75% usage indicates need for a faster cpu or additional cpu's. Then again, for all I know, you're already using SMPs.

[brityank]

You are not authorized to view this page

Ditto -- with Netscape 4.77 and IE 5.5.

Help !!

[backhoe]

"You are not authorized to view this page"

Yep, I have the same problem, too.

[Xthe17th]

Our REACH is now up to 640 (per million internet users). WOOHOO and BUMP!!! Reach is the key stat, IMHO. Yahoo's Reach is something like 300,000 - so there's our goal. Kudos for the great work Robinsons.

[Admin Moderator]

You are not authorized to view this page

Fnord.

[error99]

Thanks for everything, John !
Hope you are getting enough sleep during this !

[Congressman Billybob]

Thank you very much for translating just a few of the high points in JimRob's analysis, into actual English that us tech-dumb Freepers can understand. I do understand the benefits to FR of the compression, to stay within the bandwidth cap.

Especially good is the number of new sign-ups. FR has been mentioned in a number of lamestream media articles of late. Though most of those have been slams, not praises, both generate traffic that takes a look-see at FR. And some of those stop and stay. One more example of Mayor Curley's comment, "Say what you want about me, just make sure you spell my name correctly."

Congratulations, JimRob, and all.

Congressman Billybob

Latest column, not yet up on UPI, and FR, "Don't Trust My Words"

Latest book(let), "to Restore Trust in America."

[eureka!]

John, kudos and thanks for the job you do here. I have the stats link in my profile and check it out every so often. I noticed that signups have tripled recently. Hopefully, all the new Freepers will open their wallets come the next FReepathon fundraiser. *ducking for saying the dreaded words*

[RottiBiz]

Same thing happened to me. Click the red "source" at the top of the page and you will get this:

http://www.freerepublic.com/usage/awstats/

Worked for me.

[Seeking the truth]

I think John is saying "We're doing something good".

[John Robinson]

Oops! I posted the wrong URL. I corrected it in the article body, here it is.

I have two versions of that report. One has administrative information (IP addresses, etc.) At least I now know the access controls work.

[BureaucratusMaximus]

How about you come up with some new stats like:

Number of ZOT posts

Number of ZOT posts Zotted

Estimated time from the initial ZOT posting to when the post is ZOTed

Also...I havent seen the Viking Kitty pic in a while...and I'm quite depressed LOL.

[HairOfTheDog]

Due to brain damage, some browsers do not work properly or at all when viewing a compressed page, and therefore must not be sent compressed pages.

I hope I am not working on one of the brain damaged browsers!

[txzman]

Amazing Stats. And a great reminder that people need to help support FR with MONTHLY payments.

Liberty ain't cheap!

[John Robinson]

Dual CPU, dual server. We have two web servers. They're reached through a reverse proxy. The proxy does the compression. It really is a sight! The proxy currently reports a .122% CPU load at 57 requests per second average. And since most pages are generated and dumped into the kernel buffer to be spoonfed to the network (and the file descriptor handed off to a third party for lingering close) the web daemon is quickly freed from the request. During heavy load the daemon often only has a few requests being served at any moment in time.

(But of course, as I sit here bragging about how well it is running, it all falls down. Oh!)

[nicmarlo]

ty......I'm glad to know it wasn't just ME that was "unauthorized," lol! : )

[nicmarlo]

That's the most interesting link of all (the one you fixed), IMO.....thanks for your post and fixing the link, John....

[TontoKowalski]

New signups on Thursday: 350...the fourth highest total behind 3 days in the timeframe just after the 2000 elections.

Welcome, newbies! Glad to have you!

And now, we consecrate the bonds of obedience.

[FourtySeven]

Translation, for me: Rowr-rowr-rowr-rowr rowr, rowrrowr, rowr, rorrworwor. (like from the Charlie Brown cartoons, when a teacher or other adult would talk to one of the kids)....hehe

I'm just glad you understand it....keep up the good work Misters Robinson!

[Tamzee]

Wonderful to see :-)

[Salvation]

BTTT!

And congratulations on the new server working so well.

[Alas Babylon!]

WAY KOOL! You got a server, or system of servers, that's for sure!

[PA Engineer]

Thanks and bump.

[John Robinson]

grania--
It was removed because of a severe performance penalty due to each hit causing a database update. I have, in theory, a better way of doing it now, I'd like to try it again later.

FreedomPoster--
I would guess, from the telco's perspective, voice and digital circuits are similar if not one-in-the-same, and the pricing would probably be similar as well. I don't really know, I have neither. We're colocated with Verio in San Jose. We don't need no stinkin' CSU/DSU or other telco headaches.

tictoc--
HTTP 1.1 is probably the default on capable browsers. I haven't ever needed to switch it on, but somebody may. Thanks for pointing that out.

---

It's been a lot of fun getting this system tuned to handle the war time traffic. I've learned much about the guts of the technology we use here. I cried when I realized there was nothing left to tune. :-D We need more traffic! More tuning! MORE TUNING!

[tictoc]

Apparently mod_gzip has been around for a while, but many websites have not yet implemented it.

This page -- Making your website super fast-loading -- was not too far above my head, and it contains links to more detailed information.

[new cruelty]

Thanks for the stats. I enjoy a good read!

[John Robinson]

HTTP compression is a thorny issue. So far I've had only one report of a reader unable to visit. OTOH, without it, thousands would be unable to visit. It's time to upgrade those browsers.

[John Robinson]

Now:

Sustained bandwidth of 8 Mbits, peaking at 10 Mbits. 70 requests per second, often seeing 80 and 90 requests per second. No sweat. Give me more!

[backhoe]

Say, that's gorgeous! Thanks!

[Billthedrill]

John, congratulations on running the state-of-the-art world news distribution process, period. This is history in the making. Hang in there!

[Bob J]

John, is that 1 million uniques for Feb for the whole month or unique visitors counted each day?

[John Robinson]

It is the unique visitors for the entire month, February had more than one million.

I should warn you, "visitors" by definition of web statistics programs is the number of distinct Internet addresses hitting the website. There are a large number of reasons why such numbers do not directly translate to actual people, it is possible for one person to use many Internet addresses, and it is possible for one Internet address to service many people. See: Lies, Damned Lies, and Unique Visitors. It is a useful number when compared against other websites, though.

[gcruse]

Got it.
In the case of Netscape 7, go to Edit Preferences, Advanced, HTTP networking. My HTTP 1.1 was selected by default.

[gitmo]

the server was falling over as network congestion backed up through the kernel and into my application-- what a mess that will make!

That's why you should always keep the bit bucket handy.

[Ernest_at_the_Beach]

We need more traffic! More tuning! MORE TUNING!

It's coming!

Great job John!

OFFICIAL BUMP(TOPIC)LIST

[CheneyChick]

Cool nerd talk!

Cheers, CC :)

[ChefKeith]

You forgot the "secret handshake"

[my_pointy_head_is_sharp]

bump