Free Republic
Browse · Search
News/Activism
Topics · Post Article

Skip to comments.

New Numbers for the Curious (or FR Traffic Stats)
Free Republic Website Usage ^

Posted on 03/21/2003 3:05:06 AM PST by John Robinson

Edited on 03/21/2003 7:45:49 AM PST by John Robinson. [history]

For those that love numbers, I have three links to Free Republic usage statistics:

Caveat:

  1. I recently upgraded the program (AWStats) used to generate the Website Usage pages. The old pages, though built with the new program, utilized the old format database which did not provide some fields. Those fields will be 0 or not present on the older reports.

  2. For some reason the reports switched to GMT some months ago, I switched back to PT this month. Each switch fouled the "Hours" graph for the month of the switch.

  3. We lost our senses Mar 12-14, or rather, the stat program forgot how to tell the difference between a Hit (any request to the web server) and a Page (a request for a HTML page, or similar page of content); all hits were erroneously counted as pages, the actual number of pages for that time period is actually quite a bit less.

  4. And, on Wednesday Mar 12th, I enabled HTTP compression, practically halving our bandwidth usage! Just in time for the overwhelming load that day!

  5. The Daily Stats page is clearly showing signs of age. It no longer collects the "Front-Side" and "Back-Side" statistics. However, you will find "Front-Side" had become a duplicate effort, see the Website Usage pages to continue on.

  6. Numbers on the Daily Stats page and the new Website Usage pages are generally an even comparison. However, consider the Daily Stats pages are generated on daily GMT time boundaries, and the Website Usage pages are generated at PT (or GMT--oops) boundaries.

And now the ramble.

One thing I don't yet have for readers is a report of the actual Mbps that flows through our Internet connection. Leading up to the war we've routinely saturated our link, that's 10 Mbps, or 10 million 1's and 0's per second. Those numbers are very interesting as well!

As we reach our bandwidth limit, the network link becomes saturated and we begin to slow down. Last Wed (Mar 12) was the breaking point, the server was falling over as network congestion backed up through the kernel and into my application-- what a mess that will make!

Foreseeing impending trouble days before, I had previously installed a little gadget called mod_gzip. The mod_gzip is an Apache webserver extension that compresses HTML pages on the fly. After an abbreviated test run of just a few days I enabled mod_gzip to bail me out of this crisis.

My savior! The compression is highly effective, reducing pages by an average of 79% the original page size. However, the true savings are closer to 50%. Whether the module is used at all is dependent upon the browser used to view the page. Due to brain damage, some browsers do not work properly or at all when viewing a compressed page, and therefore must not be sent compressed pages. And images are not compressed at all-- mostly because images are already highly compressed, but partly because of even more brain damage in even more browsers.

50% isn't bad, not bad at all. We've now effectively doubled our bandwidth capacity. Have we tamed the traffic? Mostly. Bandwidth usage is now routinely in the 5-6 Mbps range, which is effectively 10-12 Mbps. Thursday morning from 7am-1pm PT we were in the 7-8 Mbps range (14-16 Mbps!) and often times pushing the ceiling, pushing our full link capacity of 10 Mbits hard for extended periods of time. That's 20 Mbits of bandwidth, 12 T-1 lines, or nearly one half T-3. Simply amazing!


TOPICS: Free Republic; Miscellaneous
KEYWORDS: adminlectureseries; faq; techindex
Navigation: use the links below to view more comments.
first 1-5051-53 next last

1 posted on 03/21/2003 3:05:06 AM PST by John Robinson
[ Post Reply | Private Reply | View Replies]

To: John Robinson
I don't understand any of that, but I know I am glad you do.
2 posted on 03/21/2003 3:07:33 AM PST by Bahbah (Pray for our Troops)
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
I'll proofread that later. I'm going to bed.
3 posted on 03/21/2003 3:07:34 AM PST by John Robinson
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
on Wednesday Mar 12th, I enabled HTTP compression, practically halving our bandwidth usage! Just in time for the overwhelming load that day!

Wow, and we did not have to download a WinZip file to do this?

4 posted on 03/21/2003 3:11:54 AM PST by LowOiL ("I would rather have a German division in front of me than a French one behind me" -Gen. Patton)
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
There used to be information about how many people were logged in to FR at any given time. It was kind of a neat thing to know.
5 posted on 03/21/2003 3:17:47 AM PST by grania ("Won't get fooled again")
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
Thanks for all you do. Love seeing the techie minutia every now and then.

Doesn't a T-3 get cheaper than all the separate T-1's at some point, less than a T-3's worth? I know it does for voice, but I guess it might not make as big of a difference for an Internet connection.
6 posted on 03/21/2003 3:21:50 AM PST by FreedomPoster (This Space Intentionally Blank)
[ Post Reply | Private Reply | To 3 | View Replies]

To: John Robinson
From the posting log stats:

Distinct Posters on Thursday: 4,358...a new record topping the previous high of 4,313 from the day before.

Total articles + replies on Thursday: 35,055...a new record topping the previous high of 31,153 set on Tuesday.

Articles posted on Thursday: 1,222...not a new record, second most ever to the 1,301 posted the day after the 2000 elections.

Replies posted on Thursday: 33,833...a new record, topping the previous high of 30,102 on Tuesday.

New signups on Thursday: 350...the fourth highest total behind 3 days in the timeframe just after the 2000 elections.

Busy busy busy

7 posted on 03/21/2003 3:23:00 AM PST by Admin Moderator
[ Post Reply | Private Reply | To 1 | View Replies]

To: madfly; Ernest_at_the_Beach
fyi
8 posted on 03/21/2003 3:29:02 AM PST by Free the USA (Stooge for the Rich)
[ Post Reply | Private Reply | To 7 | View Replies]

To: John Robinson
On-the-fly compression works only if it also is enabled in the "client browser" (the browser software that people use to log on to FR).

I believe that in Internet Explorer 6, this feature is already turned on by default.

Here is how to check:

Go to Start+Settings+Control Panel, then double click the Icon that says "Internet Options." Now click the tab that says "Advanced" scroll down until you see "Use HTTP 1.1"; make sure there is a check in the box and press ok. Press ok again to close Internet Options, and close any open IE windows. When you launch IE again you should see that surfing is much faster than it was before.

In addition, if you are using a proxy like Proxomitron (Proxo), WebWasher, etc., you should also turn on "Use HTTP 1.1 through proxy connections" by putting a checkmark into the box next to it.
9 posted on 03/21/2003 3:32:19 AM PST by tictoc
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
FANTASMAGORICALLY WONDERFUL!

GREAT WORK JOHN. THANKS TONS!

GOD'S BEST TO THE PRES AND THE TROOPS AND YOU, THE FAMILY, AND OF COURSE, FR!
10 posted on 03/21/2003 3:32:45 AM PST by Quix (MARCH BIBLE CODES DIGEST LATEST RESEARCH COMPARES WAR AND PEACE VS BIBLE W SURPRISES 4 BOTH SIDES)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Admin Moderator; Jim Robinson
A fantastic job! What a resource. Hard to believe there will ever be a Democrat elected to any high office with the info Freepers are able to gather - heh heh.
11 posted on 03/21/2003 3:47:34 AM PST by Peach
[ Post Reply | Private Reply | To 7 | View Replies]

To: John Robinson
Free Republic Website Usage NEW!

You are not authorized to view this page

So, what did I do wrong JR?? : )

12 posted on 03/21/2003 3:56:17 AM PST by nicmarlo (** UNDER GOD **)
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
How's your cpu load? Usually enabling heavy compression will further tax a cpu. Anything over a consistant 75% usage indicates need for a faster cpu or additional cpu's. Then again, for all I know, you're already using SMPs.
13 posted on 03/21/2003 3:56:53 AM PST by Alas Babylon!
[ Post Reply | Private Reply | To 1 | View Replies]

To: nicmarlo; John Robinson
You are not authorized to view this page

Ditto -- with Netscape 4.77 and IE 5.5.

Help !!


14 posted on 03/21/2003 4:08:00 AM PST by brityank (The more I learn about the Constitution, the more I realise this Government is UNconstitutional.)
[ Post Reply | Private Reply | To 12 | View Replies]

To: brityank
"You are not authorized to view this page"

Yep, I have the same problem, too.
15 posted on 03/21/2003 4:19:25 AM PST by backhoe ("Time to kick the tires & light the fires-- Let's Roll!")
[ Post Reply | Private Reply | To 14 | View Replies]

To: John Robinson
Our REACH is now up to 640 (per million internet users). WOOHOO and BUMP!!! Reach is the key stat, IMHO. Yahoo's Reach is something like 300,000 - so there's our goal. Kudos for the great work Robinsons.
16 posted on 03/21/2003 4:49:43 AM PST by Xthe17th (FREE THE STATES. Repudiate the 17th amendment!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: nicmarlo; John Robinson
You are not authorized to view this page
Fnord.
17 posted on 03/21/2003 4:54:36 AM PST by Admin Moderator
[ Post Reply | Private Reply | To 12 | View Replies]

To: John Robinson
Thanks for everything, John !
Hope you are getting enough sleep during this !
18 posted on 03/21/2003 4:56:06 AM PST by error99 ("I believe stupidity should hurt."...used by permission from null and void all copyrights apply...)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Admin Moderator
Thank you very much for translating just a few of the high points in JimRob's analysis, into actual English that us tech-dumb Freepers can understand. I do understand the benefits to FR of the compression, to stay within the bandwidth cap.

Especially good is the number of new sign-ups. FR has been mentioned in a number of lamestream media articles of late. Though most of those have been slams, not praises, both generate traffic that takes a look-see at FR. And some of those stop and stay. One more example of Mayor Curley's comment, "Say what you want about me, just make sure you spell my name correctly."

Congratulations, JimRob, and all.

Congressman Billybob

Latest column, not yet up on UPI, and FR, "Don't Trust My Words"

Latest book(let), "to Restore Trust in America."

19 posted on 03/21/2003 5:05:19 AM PST by Congressman Billybob ("Saddam has left the building. Heck, the building left the building.")
[ Post Reply | Private Reply | To 7 | View Replies]

To: John Robinson
John, kudos and thanks for the job you do here. I have the stats link in my profile and check it out every so often. I noticed that signups have tripled recently. Hopefully, all the new Freepers will open their wallets come the next FReepathon fundraiser. *ducking for saying the dreaded words*
20 posted on 03/21/2003 6:02:01 AM PST by eureka! (Bless our Troops and Allies.........)
[ Post Reply | Private Reply | To 3 | View Replies]

To: backhoe; brityank; nicmarlo
Same thing happened to me. Click the red "source" at the top of the page and you will get this:

http://www.freerepublic.com/usage/awstats/

Worked for me.
21 posted on 03/21/2003 6:34:32 AM PST by RottiBiz (If everyone gave just a few dollars a month, we'd never have to hold another FReepathon!)
[ Post Reply | Private Reply | To 15 | View Replies]

To: Congressman Billybob
I think John is saying "We're doing something good".
22 posted on 03/21/2003 6:39:36 AM PST by Seeking the truth (I'm going on the FRN Cruise - How about you? - Details at www.Freerepublic.net)
[ Post Reply | Private Reply | To 19 | View Replies]

To: nicmarlo; brityank; backhoe; Admin Moderator
Oops! I posted the wrong URL. I corrected it in the article body, here it is.

I have two versions of that report. One has administrative information (IP addresses, etc.) At least I now know the access controls work.

23 posted on 03/21/2003 7:50:40 AM PST by John Robinson
[ Post Reply | Private Reply | To 12 | View Replies]

To: Admin Moderator
How about you come up with some new stats like:

Number of ZOT posts

Number of ZOT posts Zotted

Estimated time from the initial ZOT posting to when the post is ZOTed

Also...I havent seen the Viking Kitty pic in a while...and I'm quite depressed LOL.

24 posted on 03/21/2003 7:54:13 AM PST by BureaucratusMaximus (if we're not going to act like a constitutional republic...lets be the best empire we can be...)
[ Post Reply | Private Reply | To 7 | View Replies]

To: John Robinson
Due to brain damage, some browsers do not work properly or at all when viewing a compressed page, and therefore must not be sent compressed pages.

I hope I am not working on one of the brain damaged browsers!

25 posted on 03/21/2003 7:58:26 AM PST by HairOfTheDog (May the blessing of elves and men and all free folk go with you.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
Amazing Stats. And a great reminder that people need to help support FR with MONTHLY payments.

Liberty ain't cheap!
26 posted on 03/21/2003 8:05:08 AM PST by txzman (Jer 23:29)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Alas Babylon!
Dual CPU, dual server. We have two web servers. They're reached through a reverse proxy. The proxy does the compression. It really is a sight! The proxy currently reports a .122% CPU load at 57 requests per second average. And since most pages are generated and dumped into the kernel buffer to be spoonfed to the network (and the file descriptor handed off to a third party for lingering close) the web daemon is quickly freed from the request. During heavy load the daemon often only has a few requests being served at any moment in time.

(But of course, as I sit here bragging about how well it is running, it all falls down. Oh!)

27 posted on 03/21/2003 8:10:27 AM PST by John Robinson
[ Post Reply | Private Reply | To 13 | View Replies]

To: John Robinson
ty......I'm glad to know it wasn't just ME that was "unauthorized," lol! : )
28 posted on 03/21/2003 8:12:52 AM PST by nicmarlo (** UNDER GOD **)
[ Post Reply | Private Reply | To 23 | View Replies]

To: John Robinson
That's the most interesting link of all (the one you fixed), IMO.....thanks for your post and fixing the link, John....
29 posted on 03/21/2003 8:17:40 AM PST by nicmarlo (** UNDER GOD **)
[ Post Reply | Private Reply | To 23 | View Replies]

To: Admin Moderator; All
New signups on Thursday: 350...the fourth highest total behind 3 days in the timeframe just after the 2000 elections.

Welcome, newbies! Glad to have you!

And now, we consecrate the bonds of obedience.

30 posted on 03/21/2003 8:18:01 AM PST by TontoKowalski (You say, "Thank you, sir, may I have another?")
[ Post Reply | Private Reply | To 7 | View Replies]

To: John Robinson
Translation, for me: Rowr-rowr-rowr-rowr rowr, rowrrowr, rowr, rorrworwor. (like from the Charlie Brown cartoons, when a teacher or other adult would talk to one of the kids)....hehe

I'm just glad you understand it....keep up the good work Misters Robinson!
31 posted on 03/21/2003 8:21:09 AM PST by FourtySeven
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
Wonderful to see :-)

32 posted on 03/21/2003 8:21:45 AM PST by Tamzee ("Sabotage" and "Charade"....both from the French language)
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
BTTT!

And congratulations on the new server working so well.
33 posted on 03/21/2003 8:25:01 AM PST by Salvation (†With God all things are possible.†)
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
WAY KOOL! You got a server, or system of servers, that's for sure!
34 posted on 03/21/2003 8:49:48 AM PST by Alas Babylon!
[ Post Reply | Private Reply | To 27 | View Replies]

To: John Robinson
Thanks and bump.
35 posted on 03/21/2003 8:55:51 AM PST by PA Engineer
[ Post Reply | Private Reply | To 1 | View Replies]

To: grania; FreedomPoster; tictoc
grania--
It was removed because of a severe performance penalty due to each hit causing a database update. I have, in theory, a better way of doing it now, I'd like to try it again later.

FreedomPoster--
I would guess, from the telco's perspective, voice and digital circuits are similar if not one-in-the-same, and the pricing would probably be similar as well. I don't really know, I have neither. We're colocated with Verio in San Jose. We don't need no stinkin' CSU/DSU or other telco headaches.

tictoc--
HTTP 1.1 is probably the default on capable browsers. I haven't ever needed to switch it on, but somebody may. Thanks for pointing that out.


It's been a lot of fun getting this system tuned to handle the war time traffic. I've learned much about the guts of the technology we use here. I cried when I realized there was nothing left to tune. :-D We need more traffic! More tuning! MORE TUNING!

36 posted on 03/21/2003 9:04:39 AM PST by John Robinson
[ Post Reply | Private Reply | To 5 | View Replies]

To: John Robinson
Apparently mod_gzip has been around for a while, but many websites have not yet implemented it.

This page -- Making your website super fast-loading -- was not too far above my head, and it contains links to more detailed information.

37 posted on 03/21/2003 9:27:44 AM PST by tictoc
[ Post Reply | Private Reply | To 36 | View Replies]

To: John Robinson
Thanks for the stats. I enjoy a good read!
38 posted on 03/21/2003 9:29:45 AM PST by new cruelty
[ Post Reply | Private Reply | To 1 | View Replies]

To: tictoc
HTTP compression is a thorny issue. So far I've had only one report of a reader unable to visit. OTOH, without it, thousands would be unable to visit. It's time to upgrade those browsers.
39 posted on 03/21/2003 9:36:55 AM PST by John Robinson
[ Post Reply | Private Reply | To 37 | View Replies]

To: John Robinson
Now:

Sustained bandwidth of 8 Mbits, peaking at 10 Mbits. 70 requests per second, often seeing 80 and 90 requests per second. No sweat. Give me more!

40 posted on 03/21/2003 9:52:58 AM PST by John Robinson
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
Say, that's gorgeous! Thanks!
41 posted on 03/21/2003 9:54:30 AM PST by backhoe ("Time to kick the tires & light the fires-- Let's Roll!")
[ Post Reply | Private Reply | To 23 | View Replies]

To: John Robinson
John, congratulations on running the state-of-the-art world news distribution process, period. This is history in the making. Hang in there!
42 posted on 03/21/2003 10:03:36 AM PST by Billthedrill
[ Post Reply | Private Reply | To 1 | View Replies]

To: John Robinson
John, is that 1 million uniques for Feb for the whole month or unique visitors counted each day?
43 posted on 03/21/2003 12:43:04 PM PST by Bob J (Join the FR Network! Educate, Motivate, Activate!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Bob J
It is the unique visitors for the entire month, February had more than one million.

I should warn you, "visitors" by definition of web statistics programs is the number of distinct Internet addresses hitting the website. There are a large number of reasons why such numbers do not directly translate to actual people, it is possible for one person to use many Internet addresses, and it is possible for one Internet address to service many people. See: Lies, Damned Lies, and Unique Visitors. It is a useful number when compared against other websites, though.

44 posted on 03/21/2003 2:09:37 PM PST by John Robinson
[ Post Reply | Private Reply | To 43 | View Replies]

To: tictoc
Got it.
In the case of Netscape 7, go to Edit Preferences, Advanced, HTTP networking. My HTTP 1.1 was selected by default.
45 posted on 03/21/2003 4:35:37 PM PST by gcruse (Democrats are the party of the Tooth Fairy.)
[ Post Reply | Private Reply | To 9 | View Replies]

To: John Robinson
the server was falling over as network congestion backed up through the kernel and into my application-- what a mess that will make!

That's why you should always keep the bit bucket handy.
46 posted on 03/21/2003 5:58:53 PM PST by gitmo ("The course of this conflict is not known, yet its outcome is certain." GWB)
[ Post Reply | Private Reply | To 36 | View Replies]

To: John Robinson; *tech_index; Sparta; freedom9; martin_fierro; PatriotGames; Mathlete; fjsva; ...
We need more traffic! More tuning! MORE TUNING!

It's coming!

Great job John!

OFFICIAL BUMP(TOPIC)LIST

47 posted on 03/21/2003 11:08:01 PM PST by Ernest_at_the_Beach (Where is Saddam?)
[ Post Reply | Private Reply | To 36 | View Replies]

To: John Robinson
Cool nerd talk!

Cheers, CC :)

48 posted on 03/21/2003 11:09:49 PM PST by CheneyChick (Lock & Load)
[ Post Reply | Private Reply | To 1 | View Replies]

To: backhoe
You forgot the "secret handshake"
49 posted on 03/22/2003 7:20:25 AM PST by ChefKeith (NASCAR...everything else is just a game!)
[ Post Reply | Private Reply | To 15 | View Replies]

To: John Robinson
bump
50 posted on 10/10/2003 12:16:03 PM PDT by my_pointy_head_is_sharp
[ Post Reply | Private Reply | To 1 | View Replies]


Navigation: use the links below to view more comments.
first 1-5051-53 next last

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
News/Activism
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson