Skip to comments.Another miserable FR Live Thread failure.
Posted on 10/11/2012 8:11:29 PM PDT by hole_n_one
click here to read article
LDOTTER NOTE: Hit again! Our apologies for last night's crash. Thousands got on anyway. Looks like the same wily Chinese hackers have targeted us again. We're not alone. We hear our friends at FreeRepublic got a DOS attack as well. Luis beats them back but they come in droves like in the old movies.
"LDOTTER NOTE: Hit again! Our apologies for last night's crash. Thousands got on anyway. Looks like the same wily Chinese hackers have targeted us again. We're not alone. We hear our friends at FreeRepublic got a DOS attack as well. Luis beats them back but they come in droves like in the old movies."
MUAWIYAH NOTE: I was on a different thread at the time the attack started ~ had just hit post on a response and was waiting for the edit to come back and waited and waited and waited and after it'd spun out over 5 minutes I knew the next words on TV would be "We turn now to this evening's debate and the candidates are coming on the stage"
Washingtonpost.com was also slow and posts to one thread were showing up on another.
So, everybody has old and dirty eqipment, or the networks tie up too much bandwith, or the Chicoms, or the porn vendors in Bylorussia or Bulgaria?
Its Called $$$$....
You want more reliability and scalability have to pay for it somehow... and it ain’t cheap.
I confess that I had a couple of shots of schnapps myself
last night during the debate. One of the great things I
enjoy about FR is the off the wall humor here.
It got a LITTLE slow at times....but, at least I could get on.
RE: post above...I’m referring to TWITTER
Typically the way heavily visited websites get more bandwidth is by a front end that sends the visitor to one of a group of virtual machines (or real ones). However in the case of FR the data viewed isn’t mostly static. People are beating on threads that they expect to see updated in real time. If multiple machines are sharing the load, they all also have to get all updates from one another and merge them. For applications like this, a big hunk of modern IBM iron with its raw, screaming I/O capability might beat a gaggle of Wintel blades running Linux. But the budget isn’t there to buy that million dollar computer.
Do you know how many freepers got drunk during the debate between G.W.B and the ketchup/ketsup queen whenever the queenie said, “I have a plan!”???
Evidently, we should have consulted you first. Oy vey. So sally my fliend. Sssshhhhh!!!
It was frustrating. I tried for an hour to post too, and missed some debate moments, which I didn’t notice until I watched it in the rerun later (west coast showed it 2x).
Were you going to yell at me or agree? Lol.
PS: Looky uppy sarcasm..., ansell 1 or 2 or 3 or....
I still keep in touch with a guy I met in a compuServe conservative chat room too. We just had our third baby last year.
(Yes, we are married... ) ;)
This might be a really dumb question from a non-techie:
How do chat rooms hold so many people in real time constantly chatting? While the chat would not be save able like FR threads, wouldn’t it be easier to get into and keep running in fast real time?
Couldn’t FR create a chat room only for special occasions? It would be up to JR to determine which events were worth opening the room for. Surely debates and election nights would be essential.
Would that help? Everyone just wants to be in a room with likemindeds at these times. We want to express ourselves and see how others are feeling. It has to be fast. But it doesn’t have to be a thread for eternity.
What about a special event chat room rather than a live thread?
You've got to be kidding me. Unlimited probably means "until you use all our capacity", 24/7 support is someone in India that doesn't speak English, and your processing power might be one core on a vitual machine.
Some years back there were some performance problems. We have the option of choosing displays of 20 / 50 / 100 / 250 items and I recall John putting in a change to cap it at 25 for the duration.
I don't know what the bottleneck is but last night was frustrating, and I finally gave up veiwing the live thread.
In the old days folks got cranky when the live threads were photobombed. That may have been due to a larger share of dial-ups than now.
Chat rooms simply echo input out to everyone that is connected. There's no record. The equivalent to an FR thread is if everyone was constantly reloading their browser and displaying everything that had been written since the post was started.
However, I think that a chat room would be a good idea. There are open source implementations out there, and a simple Java client could be downloaded into your browser. It wouldn't work for some tablets and phones, but it would be available to almost everyone else.
This is why I think the memcache would help.
I don't know exactly how their database is organized. But, if it is broken down into a row for every posting, then each of those could be cached with a unique key. That way, you can construct a thread from the posts in the cache.
Alternatively, there could be a cache entry for each posting, and the number of posts to the thread are embedded in the key. A quick check to a separate table in the cache to determine the number of postings would tell you if that cache entry would be valid, and it can be used to generate a posting. The cache entry would only become invalid when another post was added to the thread. The idea is simply to use the cache in place of a database queries, when possible.
Since it really seems to slow down during these live thread periods, I suspect what is really wrong is a table-locking constraint. If they are using MySQL for the back-end, it locks an entire table for an update -- which means that all read queries to that table have are blocked. Postgres and Oracle have row-based locking.
If that is the case, memcache would help, but only if was used it to reduce the number of read queries.
I would think between FReepers we could work something out with the yahoo alternative account or even those FReepers on FB using FB to recap and update coverage for these upcoming live thread events.
Whoever is starting the live thread that evening should include links to the alternatives in the first post. Smart FReepers will bookmark them in the event FR is unavailable.
I don’t know what Jim’s actually got going on in the guts. Whatever it is, has to ensure that writes are atomic, that nobody gets just part of a posting.
I don’t think he’s using a commercial database e.g. Oracle. That would be vastly too expensive to license, judging by the budgets we’re seeing.
If some well to do Freeper wills a million bucks to FR things would probably drastically change. At least a piece of serious hardware could be had.
Ok. That makes sense. Thank you.
Yeah, I think that's the problem. If they are using MySQL, it is known to have a performance constraint, although they may have addressed it in later released.
I dont think hes using a commercial database e.g. Oracle. That would be vastly too expensive to license, judging by the budgets were seeing.
While Oracle would be a great performer and scale well to a cluster, it's not a reasonable solution. The Freepathon budgets would have to double, at least.
Postgres is free, and is used for a lot of serious database implementations. Historically, it has been a better performer for large datasets. I don't know if that is still true in the latest versions. Both also support clustering, although it's a bit more hassle on Postgres.
Well, I’d like the chat room but I’d want it to work on an iPad!
We used to be slow but managable during elections. We were able to influence the MSM like have Rather take back his early call during the election of George Bush. We started and have unseated the media. We were effective. I was unable to connect at either debate and have little hope for the election.
I wonder what is going on, whether we are being spammed or attacked. It is disheartening since we are the watchdogs. And I treasure and value this role.
FR is more of a force in this country than most realize. Our ideas, our reactions, our discussions are closely monitored and reach the mainstream and the New Media. We have a job to uphold here, as conservatives re-make the media.
The Robinsons have crafted something that is bigger than any of us. This is not a blog, not a board, it is real-time grass roots news editorial. And it is fun! I thank God every day that we have this forum and I pray that it will become more and more effective and flexible as time goes on.
I have been here 14 years almost every day. I actively recruit younger members to the board, and nice conservative teens and twenties visit often.
I hope with all my heart that the software and hardware will strengthen and improve, so this wonderful idea will continue...
I am a flea on this board, and I am fortunate to chat with great brains and thoughtful commentators as well as some really funny people. Do we have jerks? Absolutely, myself included. Thank you for putting up with me.
Much love to all,
Saw that Lucianne was down and a couple of other places were not functioning well. Figured there was an attack. We need better security.
But the budget isnt there to buy that million dollar computer.
Or the talent to keep it running. I wonder if there isnt an interested mainframe owner who has space???
So frustrating last night to once again not be able to view the Live Thread on the debate!
I love FR but this has been going on for years and the inability to access it during important events is worse instead of better.
I agree with the others that suggest a chat room format for live events. I hope JimRob will consider it. Need something in place before the next debate and definitely before Election Night!
What puzzles me is that it wasn’t always like this. I’m pretty sure these problems only showed up a few years ago. I’ve been on FR for over a decade and I don’t remember speed being an issue back in the day. So the question is what happened?
Sharks would laugh this place out of the room! FR is useless - those servers have been coming for years, same rhetoric over and over and the problem still remains. Kinda makes one wonder where the money is going doesn't it.
What puzzles me is that it wasnt always like this. Im pretty sure these problems only showed up a few years ago.
I believe the problem started when JR stepped away from the technical operation of Free Republic a few years back.
Wish they would bring in someone to analyze the problem and fix it.
Sharks would laugh this place out of the room! FR is useless - those servers have been coming for years, same rhetoric over and over and the problem still remains. Kinda makes one wonder where the money is going doesn't it.
Well Jim, looks like you, and FreeRepublic's days are really, really numbered now (it appears that it may only have a few weeks, maybe a couple of months left).
Reading through this thread it is clear that there are folks posting that can do that FreeRepublic thingee you do faster, cheaper, and better than you'll ever be able to do it.
And many of them have apparently been hanging around at FR for years developing their plans to take the lead with their faster, cheaper, and better websites.
Hey guy, they've outed you as a bigger grifter than Clinton and/or Obama.
They will probably post the pictures they have of your new Malibu Mansion, your string of Cadillacs, Collectors cars, giant RVs, private 747, gold-plated and air-conditioned dog house, etc., on their stunningly faster, cheaper, and better new websites, for all the world to see.
They're probably also working with the major media outlets to release a brutal "tell-all" story about your greed, avarice, and wasteful spending of the vast amounts of money they are sending to you as fast as they can acquire it (the core of the story will probably point out that their family has been doing without the necessities of life because they've been sending all the money to you -- you know, the women and children hit hardest meme).
But hey, that's what's wonderful about America, eh? They'll build it, and the giant sucking sound heard around the world will be every last one of the people currently on FR, as well as those who ever did, or ever will, visit here, rushing to join up at those faster, cheaper, and better new websites.
So, sadly, it seems you might as well just give in to the plans of these new web magnates and throw up a giant flashing announcement on the front page of FR that you're going out of business.
And you need not even think about selling any of your equipment to these dynamos of the interwebs, after all, they can acquire and provide equipment at almost zero cost, and 24/7 service, with 100% up-time rates, at prices so low, it might as well be free.
It was good while it lasted Jim, but these go-getters have plans that you clearly won't be able to compete with. Their business and operational plans are so good that when you sign-up at their faster, cheaper, and better new websites, your home pc/mobile device will actually run faster than it has ever run before!
But Wait! If you sign-up in the first 24 hours, you will get a free lifetime membership (for the lifetime of their faster, cheaper, and better new websites), and 3 sets of stick-free cookware (not sold in stores, S&H extra).
Jim, don't feel too bad though, these folks are just natural-born powerhouses who will ROCK the interwebs!
Google, Twitter, Facebook, Yahoo, Apple, Microsoft better be watching their backs! These faster, cheaper, and better new websites are just going to be that good!
So, as you fade into history, knocked out of the top spot by these individuals who will change the world, just console yourself with the fact that you were privileged enough to have had them spend a little time (well, actually, years) at your useless site, before they ROCKED OUR WORLD!
I know I will be watching for the opening of their new sites -- I'm soooo excited just thinking about it!
Wow, quite a cynical post. The simple fact is that the site isn’t performing as it should. I’m sure even Jim would agree this is true. You may be okay with FR choking at important moments, but some of us aren’t. Some of us remember when FR could handle the load. There’s a difference in being a supporter and being a cheerleader. Supporters want FR to live up to its mission. Cheerleaders shift blame when it doesn’t. I’d like to see FR succeed.
Wow, what a sensitive post. Since my post was not addressed specifically to you, are you self-identifying with its contents?
The simple fact is that the site isnt performing as it should. Im sure even Jim would agree this is true.
So you have identified site performance as a problem, and you're sure Jim would agree. Well, now that's a real revelation, especially since some posters are spending some significant time pointing it out over, and over, and over, ad nauseum.
And Jim has agreed numerous times in his posts that he knows that there are problems, and that he is working with John to correct them. (You could have included him in your address line, since you are referring to him).
You may be okay with FR choking at important moments, but some of us arent.
You don't have a clue what "I may be okay with".
Also, since you are referring to "some of us" while responding to my post to someone who said FR was "useless" and inferring that Jim was using the donated money for something other than running/fixing the site, I can only assume you are in agreement with that poster's assessment of Jim's credibility and honesty.
Some of us remember when FR could handle the load.
LOL, check my signup date noob.
I guess in your world, nothing ever changes (like the Internet technical and business operating environment, increase in the number of FR visitors between now and then, intermittent technical problems, Denial of Service attacks from within and external to the U.S., etc.).
Theres a difference in being a supporter and being a cheerleader. Supporters want FR to live up to its mission. Cheerleaders shift blame when it doesnt. Id like to see FR succeed.
Oh my, what blather.
I was curious who would respond to my post. I notice (happily) that it wasn't one of the posters who regularly try to suggest something technical to try to help Jim/John in analyzing the possible problems with FR.
As I suspected, it was someone who is apparently easily "offended" that anyone would point out that there are some people who have an axe to grind with Jim, and/or are just types who love to complain, complain, and then complain over again. On, and on, and on, posting and reposting their non-useful complaints until it is enough to gag a maggot.
LOL, I'm a cheerleader and you're a supporter -- LMAOROTF.
Well, Yardstick, regardless of your post, I'm still looking forward to your creation of your newer, faster, and better website. It will undoubtedly be marvelous, and I'm sure it will ONLY be filled with supporters who will sing your praises 24 hours a day, everyone will ride the unicorns, and there will be free skittles for all participants.
Oh yes, and since you undoubtedly took critical time away from your efforts to implement your newer, faster, better website, please don't waste any more of your valuable time responding to a lowly "cheerleader for FR".
Why, I'll probably sign up at your new site, to get that a lifetime membership and the 3 sets of stick-free cookware (not sold in stores, S&H extra).
No excuses, no denials. The system is failing under peak loads. Don’t know why yet. I thought we might be able to solve it by beefing up the servers and the RAM, and raised $6,000 to do so, but guess John does not agree. He’s wrestling with some systems configuration problems at the moment and has not yet found the solution. Unfortunately, I cannot help him with these issues as they’re way over my abilities. I’ve asked him if bringing in a consultant will help, but he appears reluctant to do so. I’ve also asked him to come on to a thread to post the status and answer questions, but so far he hasn’t wanted to do that either. Hopefully he will eventually find the problem. Sorry about the slowdowns during the debates and praying we get it fixed soon. If we end up not needing the $6,000, I will offer to return it to the donors. But I’m still thinking we need to beef up the hardware.
Jim, That $6000 will be needed eventually no matter what. It’s hard enough to collect it when planning for the future, imagine trying to collect it in an emergency.
My vote would be to save it for later if not needed right away.
I have nothing but respect for you & yours and the wonderful work of love and dedication and patriotism that is FreeRepublic.
Please do not misconstrue any of my occasional gripes re system sluggishness as commentary other than technical.
The integrity and honesty of you and all those behind the scenes of this place have never been in doubt in what little is left of my mind.
(well, there may have been a couple mods i wanted to slap over the years but . . ;-)
Seriesly tho, my very best hopes to you and John and whoever else is/may be working on getting us back up to speed, literally.
Jim use the money. Get doo dads for the board.
Why was my post deleted tonight? Was I bad?
Roger that, Jim. We know you guys are trying to get this worked out. John’s forum software is the best bar none. Given what he’s accomplished there’s no loss of face in letting a consultant come in to do some housekeeping if that’s what’s needed. I have no idea what the right solution is. Something definitely needs to be done though. We’re all pulling for you.
Jim, in all the years I've been at FR, I've never known you to give excuses, nor denials. I'm not expecting to hear them from you now.
I suspect that some FReepers don't fully understand that this is your baby (and now John's as well), and just how frustrating it is to not see them quickly resolved.
I spent many years at remote locations, maintaining national defense radars, and I am well aware of the frustration of having an intermittent and/or non-repeatable failure drive one to the point of hair pulling.
On one of those systems, I arrived at day shift to find the maintenance crew had removed and replaced every Line Replaceable Unit (LRU) with one or more of the spares.
They had depleted all of the spares stock based on the Main Computer's software "instructing" them what LRU to replace.
They had followed the automated troubleshooting software's instructions throughout the night, and just before I arrived, it had gone full circle and was now instructing them to replace the first LRU it had called out at the beginning of their shift. Spares that had been called out, and subsequently identified by the computer as failed after retest, were to be found all over the maintenance area benches and shelves.
It was a nightmare, Headquarters was readying an emergency response team and transport to fly them in to "assist" us. The Site Commander was in a bad situation because, if they did have to be sent in, he was probably going to lose his Command position, and his upward mobility in rank would be forever lost.
As the Technical Services "expert", I was on the hot seat as well to find and fix the problem asap. This National Missile Early Warning System had been down for over 8 hours, and it's expected outage time was expected to be not more than 15 minutes (hey, it was an automated, multiple-stage computer-controlled, self-diagnosing and failure-unit-identifying system after all).
It took a couple of hours more to find the problem, and at first the maintenance crew I was technically guiding didn't believe the failure I led them to. After fixing what I showed them, the system was restored and operational within a few minutes. The Site Commander also wanted to be personally shown what the problem was, and some help in verbalizing the failure and fix to the General and his staff.
We just managed to restore the system in time to cancel the visit from the assistance crew (bet their families liked that because they didn't have to fly across country to help us).
As a Technical guy, it was a high stress, frustrating, and yet very satisfying experience.
So while some here are convinced that I am just a "cheerleader for FR", based on my experiences in finding and fixing hardware and/or software problems, you have, and continue to have my support.
FR is unique. For those who don't believe it, probably any other site/forum will do.
Oh yea, what was the problem I found? Two wire-wrapped power distribution standoff posts in the Radar Receiver Cabinets had gotten bent so that they were just close enough together to intermittently "arc-over", spiking and/or temporarily dropping the voltage to the LRUs below acceptable levels. This confused the computer software's automatic single-point failure diagnostics into reporting all of those (actually functional) LRUs as failures.
Keep on trucking Jim, and tell John to do the same. Hopefully you will both know if/when outside experts are required, and they will be summoned (and be of use).
That $6k can buy a lot of on-demand cloud computing. Any number of services will allow you to spin up extra capacity for peak times then shut it down.
It sounds like you are in between a rock and a hard place. As the old saying goes. Not wanting to insult John, who has over time, proven that he is a smart fellow. But as to John not wanting help at this point. Maybe Remind John that Pride goeth before the fall. Sometimes no matter how smart we are then we can’t see something that is right in front of our face. A Fresh look from someone else might discover something that was missed.
Jim, I come from old mainframe COBOL DMSII programming environment.
When we knew our hardware, disk space and database settings were adequate and still had a problem we looked to the software.
Even on good days here on FR the response time is slower than it used to be.
I am wondering if there is a programming glitch that causes each transaction or maybe just one to somehow go into an unnecessary loop (or whatever they call it nowadays) before returning the response. This would eat up your memory and cause all other transactions to wait for this one to end.
This may not be noticeable on a slow day but of course be exacerbated with heavy volume.
Just a thought from a mainframe dinosaur.
One other thought. While I know John takes great pride in his work, most times a pair of “fresh eyes” looking at your code can spot a bug that you’ve overlooked time after time.
More than an hour to go till the final debate tonight and Free Republic is barely usable.
I disagree with you on a few points:
1. Real time shouldn’t be demanded or expected. There should be cacheability for short time bursts, and since people tend to hit the same pages, the potential for cache misses should be low.
2. Transactional consistency shouldn’t be the rule here. No reason for it. Eventual is fine, and even that might be on the strict side.
3. There are plenty of improvements that can be made on the software here. I’m sure John has done an excellent job of making sure a lot of static content is cacheable, though.
There’s no real reason that the servers should be this slow, and from what John’s indicated, the problem doesn’t seem to be that the database or hardware is stressed. Synchronization points aren’t always obvious, though. In a distributed system, everything pulls on everything else, often in unexpected ways; it’s not a series of isolated components connected by a network. Could be improper backoff. Could be a case of improperly managed blocking queues. Or it could be external altogether & be DDOS.
Interesting problem to tackle, I’d love to give it a try.