Fibs DB bot concept

[gammboy]

I am starting a fresh thread to gain fresh insight into this concept, and maybe clear up some confusion over the intent.  Please comment if you have ideas, concerns, questions, etc.  

OK, based on my own ideas and helpful input from fibsters,  the current idea works like this:

A bot collects data from fibs, and logs it into a database.  A web page will be developed that will organize and display statistics and information derived from that collected data.  This web page will be "real time", meaning, the instant that an event happens on Fibs, it is available to, and displayed on the web page.

This web page might have, for instance a scrolling window with real time events, like matches ending, etc.  A "Who's hot, who's not" display showing weekly gains/slides in ratings that you can click on to drill down into details (match data, length etc.)  This website might also have a personal page that has statistics about you.  It could allow you to make notes about individual matches, etc.  You could set up customizable alerts to have the bot tell you (email you?  page you?  ICQ you?)  when a given event occurs, etc.

Access to the web page will be controlled and limited to current Fibs users.  You will grant yourself access to the page by telling the bot to set you up with a password.  The bot will assign a password for you that you can later change.  (If you choose to change it, a warning will be issued that you should use a DIFFERENT password than the one you use for Fibs.)

There could also be an interface available to other bots and client programs to make use of the historical data, as well as a bot you can ask questions of to get information.

Please post any ideas, etc.

mv flames, threats, demands > /dev/null

GB

PS  Patti has offered to include YEARS of historical data she has collected to make this a really complete archive!  I will only ask her for this data should this idea become reality.

[diane]

I am trying, but I cant think of any objections  
Sounds interesting  

[socksey]

Cool!  I'm especially interested in the Patti archives.     Will we still be able to access any shout archives?

socksey




"Madam, you have between your legs an instrument capable of giving pleasure to thousands, and all you can do is scratch it!" - Thomas Beecham (while singling out a cello player for criticism)

[spielberg]

Good information (I'm an info junkey!) would be available thru' this . To be able to do the performance graphs available in Java for other people too - yes please.

[inim]

There could also be an interface available to other bots and client programs to make use of the historical data, as well as a bot you can ask questions of to get information.

I think this idea is very promising. Having worked on bot and client code quite a bit i found that for 20% of the time you work on your application, and for 80% of the time around oddities in fibs data and the CLIPS protocol.

There are two ways to do the service you sketched, both of them justified indepdently.

1) Debugging the CLIP Stream

There are some bugs in CLIP which are mere imperfections. Think of the Spanner at the Works and the RawWho bugs. A statistics server needed to work around both anyway, just as any other bot and client has to. A technical idea to make use of this could work like this. Your server already reads the complecte CLIP stream via "watchdog". What if it offered the cleaned up fibs stream for use by clients and bots? Technically speaking it would be a stream multiplexer with some basic filter logic. I'd be glad to contribute existing Java code doing for cleaning, parsing and multiplexing the CLIP stream.

2) Extending CLIP

It's obvious that fibs can not change protocol without breaking the existing bot and client base. A 24/7 watcher service could more easily introduce new features by wrapping CLIP and then re-implementing a cleaner protocol. An example is the current "board:*" message. It is incomplete to the end that it does miss 2 important pieces of information, namely the match state and the resign state. Without this flaw, clients and bots can literally save hundreds lines of code. So if a state logic was done centrally AND new message formats (say, boardext:*") were sent from that server, a new breed of bots and clients was enabled.

To sum up, the idea is a proxy service for fibs.com which augments the current CLIP stream and/or rewrites the CLIP stream into an improved protocol.  These ideas were discussed with Patti already and she considered them promising. The main problem so far was her fear (well justified one) to change a running system (fibs)., but with a proxy like you propose (btw, got a name for your service already?) allows for more daring architectures.

Gammboy, do you think you are willing and have the ressources to plan into that direction? I share a lot of your ideas (beside logging shouts, obviously) and have quite a bit of your stuff implemented myself. All I was missing so far was a reasonbly stable and powerful hosting location.

Thus: would that be an open source project so others can contribute? Would you reserve access to your server to yourself? Will it be possible to access the complete data you record in some raw format (SQL queries or text)? And finally, what is the proposed implementation language?

inim

[inim]

Many fibsters use more than one account, e.g. to have one for public shout and one for playing a bit more silently. Being able to query "who logged in from IP w.x.y.z" would for many of them spoil this. Of course this kind of anonymity has been an illusion anyway, but by building an easy to use service the technical barrier is significantly lowered. So what's your opinion on that type of query? Has some pros (droppers and cheaters are easy to spot, abusers can easier be traced etc.) and some cons (no way left to hide anymore beside IP spoofing).

inim

[gammboy]

Gammboy, do you think you are willing and have the ressources to plan into that direction? I share a lot of your ideas (beside logging shouts, obviously) and have quite a bit of your stuff implemented myself. All I was missing so far was a reasonbly stable and powerful hosting location.

Thus: would that be an open source project so others can contribute? Would you reserve access to your server to yourself? Will it be possible to access the complete data you record in some raw format (SQL queries or text)? And finally, what is the proposed implementation language?

The hosting is a problem.  In fact, that is the biggest problem I face.   I think your ideas about providing a data service to other bots and programs is a logical extension of the ideas I have, however it is a bit beyond the scope of what I am currently doing.  Assuming an affordable host could be found, it would be trivial to implement a SQL or text query protocol.

If security of this data is important to fibs users, then a single administrator of the database is called for, and controlled access is required.  Even though the data that I am collecting is freely available for anyone else to collect as well, open sourcing the code to do so would be asking for a swarm of collectorbots possibly controlled by people less scrupulous than I.

I am currently developing this with PHP against a MySQL database.  

Many fibsters use more than one account, e.g. to have one for public shout and one for playing a bit more silently. Being able to query "who logged in from IP w.x.y.z" would for many of them spoil this.
<snip>
So what's your opinion on that type of query?

My opinion?  I think this might be another question for fibsters.  Personally, I don't care who knows what IPs I log in from.  I can see why some would.  I do think that most who would be opposed would be those who are up to no good.   You mentioned several good uses for this info.  Do these benefits outweigh the risks?  

Fibsters:  What information would you object to others seeing about you?  Ratings progress data?  Match data?  Who you played?  Average time per match?  Login/ logout times?  IP address info?  Tourney performance?  Your input matters!

GB

PS - I apologize for the delay in this reply, I spent the last week in lovely Edenton, NC implementing a middleware project.

[don]

inim's point is great.  Logging in from the same IP does not necessarilly mean the same user, but it provides an indicator.  The information is available anyway, so what could be the objection to making it easier to get?

[socksey]

I object!  No shout logs?  Then why bother?  

socksey



"There are two sides to every divorce Yours and s**thead's." - Anonymous

[spielberg]

I support sox's right to be objectionable, that's a femiale tradition, yet disagree with the need for a shout record. In terms of my particular views on data collection I do not mind any data being collected on fibs - it's my phone calls and emails that I object to being scanned tho' given the current environment I suppose that even that is necessary.

On the bot itself I agree with don (for once) that ISP captures correlated to user names would be useful yet feel that the more "interesting" data (eg change in ratings , frequency  of play, average duration of matches etc.) particularly for any user would be more popular.

In terms of the implemenatation it's up to you obv. gammboy whether you wish to make it open source tho' I personally would prefer you not to -  the fibs server could end up swamped with bots and tho' Patti could configure monitor accordingly this too would take processing power and slow down an already stretched (at times) server......my ideal would be solely access to the end data. I don't really care in what format that comes indeed (tho' OpenOffice offers other formats) sometimes feel nostalgic about csv's.

[Zorba]

Fibsters:  What information would you object to others seeing about you?  Ratings progress data?  Match data?  Who you played?  Average time per match?  Login/ logout times?  IP address info?  Tourney performance?  Your input matters!

All this data is okay with me. Login/logout times may not be very useful, maybe it's better if that was left out, but I don't mind personally.

What I would object to, but which isn't in the planning AFAIK, is a bot that starts watching my matches without me asking for something like that.