Tourneybot Frozen

[amarganth]

Hi all

Yesterday, it was the second time, that TourneyBot does not respond to commands, sent to it. It happened to all people, not only to me.

Last Sunday, I received the result of all sent commands after some minutes.

Yesterday, I got no answer of it for about 30 minutes. After that time, someone restarted TourneyBot.

We played the tourney under TBot. (Thanks diane!)

Do you have an idea, what the problem is, MadMatt?

amarganth

[webrunner]

I just restarted tourneybot.
Hope that helps.

[amarganth]

Thanks, webrunner.

But you corrected only the symptom, not the reason.
There is a reason, that TourneyBot freezes.

[MadMatt]

I don't know why this started happening all of a sudden. I haven't changed anything recently and before I only did minor changes that don't really affect the code that much, like added the bagolympic type.

Webrunner, have you changed anything on the server? Like, updated ruby or something else?

I've enabled logging, maybe we'll see some exceptions in the logs next time it happens.

[webrunner]

I haven't changed anything that has to do with the bot.
I have updated a lot on the server, but i do that all the time.

[amarganth]

TourneyBot was frozen, today, at about 11am thru 12am. After a short while, it answers to all stacked commands.

Can you see something in the logs, MadMatt?

[Anakin]

I was trying to host Bagolympic today between 9:30 - 10:30am utc, but TourneyBot will not respond to any commands.  

Why is this happening?  

[Tomawaky]

You can try TBOT when Tourneybut is out of order

[webrunner]

I don't host Tourneybpot anymore.
Ask Madmat for help.

[diane]

I am pretty sure this is being done maliciously, quite what the objective is, is beyond me.  What it is doing is annoying, for sure, but it isnt going to stop me running tournaments, or coming to FIBS.  I will be doing them the old fashioned way from here on in - since this person seems pretty determined to keep being a pain in the ass - bans are much easier to handle this way too  

For a long term solution - the only thing I can see as a possible solution, is to have more copies of the bot running and/or available - since it seems to take some time and effort to knock them down - and if a TD can start up their own bot to run the tournament they are running - it should last until the end of the event.  The bots that we have running are not manned even close to enough - so they cant be rebooted as often as is necessary - and cannot be made operational on the time scale needed - so as long as this person / people are hell bent on spoiling the tournaments - we either need to remove the technology and do it by hand - or find a way around the situation where only two copies of the bot are running - and by people who arent watching over them when needed.

[Tomawaky]

As I already posted here.

If Tourneybot or Tbot are not available, I will not host anymore Online Tourney on fibs.
I do not host many of them, but I took really no pleasure to do it.
I rather prefer playing, especially when I know Servers where you can manage tourneys without any TDs.

I only do personnaly one bago a month cause we are not enough TD and I think I must take my part. But if it must be done manually, it's for me a clear Double / Pass  

Bot help us a lot and there is no many thing to do, but doing more is too much for me.
I already spend enough time for the league and offline tourneys...

[amarganth]

I understand that, and it's also my solution. If TourneyBot or TBot are answering, I will run my tourneys, but without bot help, it's too much (unnecessary) work.  

[diane]

I am disappointed you both feel this way, this is a deliberate attempt by a person to stop tournaments from running.  I wont give in to it - and I would hope others would try to give it a go, it really is very simple - and with a few tools I think it can actually be easier than using the bots.  We have a few basic tools for running Fridays3, which we would be willing to share - and honestly - there is nothing more to it than some cutting and pasting (course you need an interface that does that).  Please be at least willing to try - if you still think it is too much work - I guess we give in to the 'terrorist'    

[amarganth]

Ok, Diane. I will give it a try. You know my mail address.

I'm afraid, if I will start a tourney manually/offline, that there will (sure!) be some persons, that accept some "bye"s in the first round, or some opponents only with some nervy comments.

[diane]

Using these tools, the positions are randomised and byes given in much the same way they are with the bot. People have always made jokes about that - at the end of the day - you laugh and ignore it.  The odd few will always find things to moan about - the majority will appreciate you have nothing to gain by 'fixing' positions.
The only real difference you may notice is that running a tournament manually does mean you cannot play in it yourself.  You need to be available to send out match pairings and to take match results.  Of course this means you will have to be slightly more altruistic than for a regular tourneybot tournament, but it also means - since you arent playing - you cant be accused of giving yourself a bye etc.

[Tomawaky]

The big Advantage of the bots is the automatism = no discussion.

When you start doing things manually, there always would have discussion about cheating.

And don't reverse things diane. There is no 'terrorist' !
Simply not enough people who want to be involve in Fibs in the long time.

We all saw many people who get involved in some kind of projects which improve Fibs. But with time, they give up for many reason that we can understand....

Another thing. I started the league more than 4 years ago and it is still up. Really sorry to disappointed you Diane but I prefer to do what I am able to do long rather than start something that I will give up by the time.

[Tomawaky]

The only real difference you may notice is that running a tournament manually does mean you cannot play in it yourself.

And that's what I explained upper. I really prefer playing than hosting.
I do not enjoy hosting as some other do.

[diane]

Simply not enough people who want to be involve in Fibs in the long time.

We all saw many people who get involved in some kind of projects which improve Fibs. But with time, they give up for many reason that we can understand....

This is very true - and it is a shame that someone deliberately knocking the bots offline to further discourage those few volunteers should be successful    That is my main concern - it is highly likely that if these attempts fail - the person may give up - and tourneys can become easy again - that is my ultimate aim - not that more people give up and move on.  Obviously it is up to each TD how they wish to handle the situation.  
I understand and appreciate what you do Tomawaky - and know that you already give much time - it is the others who maybe can cope with a tiny bit extra work that I wish to encourage to hang on in there.

[socksey]

and it is a shame that someone deliberately knocking the bots offline

Is this an established fact or is this a supposition?  If this is the actual case, how do we go about finding out who the culprit is and neutering him/her?  

socksey



"Force, no matter how concealed, begets resistance." - Lakota Proverb

[diane]

It is a fact - but I am not going to give them the satisfaction of naming them, they can own up and take the rap, or remain an unknown pest.  There is no way to neuter them - every escalation causes a counter escalation, this is why I am just moving outside the vicious circle.  I expect the next step is to knock me offline while I run tournaments, but that is easily dealt with.  To be honest I have no idea what other tricks can be done to make life awkward for those trying to be positive, but I will plod on attempting to provide some fun for fibsters for as long as it is possible.

[MadMatt]

and it is a shame that someone deliberately knocking the bots offline

Is this an established fact or is this a supposition?  If this is the actual case, how do we go about finding out who the culprit is and neutering him/her?

I'd be interested in finding out how this works as well, so that I can plug the hole that enables this person to knock TourneyBot off. If it's true, then it's quite obviously a serious bug in TourneyBot. If you have information, please share it.

[burper]

and it is a shame that someone deliberately knocking the bots offline

Is this an established fact or is this a supposition?  If this is the actual case, how do we go about finding out who the culprit is and neutering him/her?

I'd be interested in finding out how this works as well, so that I can plug the hole that enables this person to knock TourneyBot off. If it's true, then it's quite obviously a serious bug in TourneyBot. If you have information, please share it.

Have you taken the time to look at the logs at all MM? I would think they would reveal something, wether it be external or internal.

[webrunner]

I never had any problems when the bot was on my server.
It never stopped as far as i know.
Maybe that is because i had a firewall installed.
Or is it knocked out thrue a glitch in the code?

[OFFTOPIC]
Hi Burper, you back again?
[/OFFTOPIC]

[juggler]

Have you taken the time to look at the logs at all MM? I would think they would reveal something, wether it be external or internal.

Hi burper, hi all,
I just have browsed through the logs and not seen anything at all. The tbot log of FIBS communication ended in mid-word, the process was still alive (that's why the shell script wrapper didn't restart it, unfortunateley), but tbot was not logged in any longer. It's conceivable that certain inputs (typically excessively long lines etc.) could make it hang, but that's just an assumption. If you have specific ideas let me know -- I may have a look as soon as I have played enough bg for the day, which may be tomorrow, next week or next year.

I have just restarted it so that it is available again. "Sorry for the inconvenience." ;-)

Regards,
Peter aka the juggler

[diane]

I never had any problems when the bot was on my server.
It never stopped as far as i know.
Maybe that is because i had a firewall installed.
Or is it knocked out thrue a glitch in the code?

If you check the date of the first post on this thread - it is Oct 19th 2005, this post looking for a new home for TourneyBot was made in November last year - and the actual move took some time.  This means it did happen whilst webrunner hosted it - or am I missing something?  
The point I am making is that exactly the same pattern is occurring independant of where the bot is geographically, the firewall protection it has, what type of machine it is held on etc (all a bit over my head). These pieces of information should be considered in root causing the problem I think.  
The other good indication that this is a deliberate act - if you think about it, is the time the bots hang.  They can be fine for days - right up until you need them - then the problems start, often once a tournament has begun - causing much confusion and giving those ever grumbly fibsters loads more to grumble about.
I hope the issue can be resolved - but I cant help thinking we are dealing with an ongoing issue - you fix this - the person will find another hole - for what reason, well who knows?  This is why I suggested TDs have an idependant copy of the bot they can run as and when needed - but after talking to juggler - this doesnt seem particularly feasible - since it comes with a whole load of 'baggage' that I dont particularly understand.   The other solution is that they are hosted by someone who can be there, if not 24 hours a day, then at the times of tourneys to do the restarts - this seems the least sensible solution.  

[juggler]

[...] The other good indication that this is a deliberate act - if you think about it, is the time the bots hang.  They can be fine for days - right up until you need them - then the problems start [...]

Hi Diane,

the start of a tourney coincides with the bot being used after some idle time -- now since an idling bot is unlikely to crash spontaneously, the pattern you describe is well compatible with a simple program flaw.

Kind regards,
Peter aka the juggler

[diane]

For sure - but they dont crash in normal usage - ie running a tourney in the normal way.  They also dont go down in sponteneous tournaments - just the scheduled ones.  At first they would be fine right up until you came to use them - ie before you typed in the first commands - now it is obviously more fun if the tournament gets underway before it crashes - that way there is more confusion and misunderstandings - with matches started not counting etc.
There have been genuine reasons for the bots going down simultenously - but the symptoms are very different.

[amarganth]

Juggler wrote:

The tbot log of FIBS communication ended in mid-word, the process was still alive (that's why the shell script wrapper didn't restart it, unfortunateley), but tbot was not logged in any longer.

Do you *flush* the buffer after writing an entry to the log? If not, I would try it like that. May be the entry is written logically, but not saved to the log file.

[webrunner]

Diane you are right, i didn't look at the date.
Still it is a new problem. The bot ran for years correctly on my server.

Maybe it is an idea to do an automated restart every 8 hours. That way it will never me down longer then 8 hours. This should me very easy to do. (using a cron job)

BTW, i still haven't found any host for the mgnu bots.  <_<    

[diane]

Maybe it is an idea to do an automated restart every 8 hours. That way it will never me down longer then 8 hours. This should me very easy to do. (using a cron job)

OOoohhhh that sounds like a plan!! Even if it came in the middle of a tourney - it would be better than losing it altogether!
Is that easy to do?

[amarganth]

Sorry, if I write it that direct:

The problem lies inside the code (Ruby).

Restarting the bot is fight against symptoms, not the cause.

[socksey]

Do you *flush* the buffer after writing an entry to the log? If not, I would try it like that. May be the entry is written logically, but not saved to the log file.

Amarganth,

What does this mean and how is it done?

socksey



"Our job is to give people not what they want, but what we decide they ought to have." - Richard Salent, Former President CBS News.

[amarganth]

It's for the programmer. Just an idea to find the last action before the bot freezes. I hope, MadMatt or juggler can change that in the Ruby-Code of TourneyBot or TBot.

In (nearly?) every programming language, if the program writes data to a file, the system answers with "ok", even if the system havent written the data to the file. The system usually writes data in buffers, because it is much faster, to write a full buffer to disk than every byte separately, in the worst case.

If a program interrupts (or freezes and must be cancelled manually), it is possible, that the last buffer isn't physically written to disk, only logically. And so the last data is lost. So my idea is to flush the file after every write to the log file, just for debugging. May be, the admin can find more, if the TourneyBot freezes the next time.

I don't know Ruby, but I found a description of "flush" under http://www.rubycentral.com/book/ref_c_io.html: the command exists.

[diane]

Thanks - thats all clear now    Still - just so long as someone who can use it understands it    

[amarganth]

really  , diane?    

[socksey]

  Thanks for responding, but I think I'm sorry I asked!  

 

socksey



"Hatred is toxic waste in the river of life." - Micron

[diane]

really  , diane?  

It is ok - I am a mere mortal - not a coder - I didnt expect to understand    

[webrunner]

I agree that restarting the bot is not a solutiuon for the long run.
That wasn't my intention. But it is a short-term fix.

[MadMatt]

Have you taken the time to look at the logs at all MM? I would think they would reveal something, wether it be external or internal.

Hi burper, hi all,
I just have browsed through the logs and not seen anything at all. The tbot log of FIBS communication ended in mid-word, the process was still alive (that's why the shell script wrapper didn't restart it, unfortunateley), but tbot was not logged in any longer.

Yeah, this is exactly what I see as well. If I have the time I'm going to look into it this weekend. I'll try to improve the logging facility so that we can maybe get a better read of the situation.

Someone also sent me a command that knowingly crashes tourneybot, I'll look that up as well.

[MadMatt]

This is why I suggested TDs have an idependant copy of the bot they can run as and when needed - but after talking to juggler - this doesnt seem particularly feasible - since it comes with a whole load of 'baggage' that I dont particularly understand.

There's no baggage involved, it's very simple to install and use it, which was one of the goals. I opted for a neat library that does permanent storage instead of perhaps something more powerful like a proper SQL database for that very reason.

All you need to do is install a ruby interpreter, which is just like installing any other application, download TourneyBot and run it. Of course, you'll prolly want to modify settings.rb to make yourself the admin and such, but since that file is heavily commented I don't see why that would be a problem.

[MadMatt]

Okay, I coded in a better logging facility that captures and saves all input and output to and from the bot. Maybe this will reveal something.

I also fixed a pretty major bug that crashed the bot on a certain kind of tell, though since the bot crashed properly (and reconnected after 30 seconds) I'm pretty sure that's not how whoever's crashing the bot does it.

To those of you who are running your own copies of the bot, you should update your versions from  the CVS repository and look at settings.rb to configure logging. Be advised though that the logs can fill up quickly, so you might want to check and clear them every few days or such.

Hopefully this will shed some light on the situation.

[don]

I am amused that people bother to use TourneyBot to harass me when I'm not even logged in, as in the following message from a well-know multiple-ID user.

From TourneyBot (Jan 28 08:36 PST): You have been banned from Tourney #1643,   by the order of ***a director***.

I substituted "***a director***" so I wouldn't be accused of personal flames here.  I'm merely citing yet another tool perverted by those who pursue flames.  The fact that I was left this while not logged in shows the abuse level I have to put with from about half-a-dozen users of FIBS.  I pity those folks.

--
don

[MadMatt]

It does seem that some people have taken up creating tourneys solely for the purpose of banning one or two individuals, after which they delete the tourney. Some of these people have now been banned from TourneyBot.

I'll change it so that you're only sent a tell when you're banned from a regular tourney, which means you won't see anything if you're not online at the time.

Also, since some people apparently didn't understand my previous posts, TourneyBot now logs EVERYTHING. Every tell you send it, every message or tell it sends you, everything that comes its way. It's true, I have neither the time nor the will to police the bot and sift through the logs, but it's certainly possible to do. If this becomes too much for me to handle, I'll simply take the bot offline, it's nothing to me if the bot runs or not, I don't have time for FIBS anymore and it's public domain code anyway - anyone can run their own copy.

[adamosad]

Wow what a huge post...  I wish Madmatt to find the bug soon...