RepBot/SavedGames

[wintom]

For those who are too lazy to build there own JavaFibs/RepBot toolbar, here it comes:

• The icons
• The buttons file

The icons are made by FAMFAMFAM.

Visit my blog for a detailed usage description (in german).

There you can also download a script (Java) that automatically retrieves the saved games list and puts everybody that exceeds a certain threshold on the villains list.

hth,
Thomas

[stog]

thx wintom

how do we install the villain's script?

[wintom]

Simply put it into the user-folder in JavaFibs. In order to run it go to start->run and type "cmd". Navigate to the user-folder by typing "cd <folder>" in the new window and start java with "java FibsVillains <threshold>".

Thomas

[wintom]

Ok, this "start->run" thing is for windows... On OS X open the terminal and start with "cd <folder>", where <folder> ist the name of your personal user folder (e.g. /Applications/JavaFibs2002/user/<username>) and <threshold> is the number of saved games somebody needs in order to be added to the villains list. So for 20 games you would type: "java FibsVillains 20".
While doing all this JavaFibs should not be running, because it would delete the list when closed.

Thomas

[stog]

thx i now have a bag of villains (>25 saved!

[spielberg]

Thanks wintom - I'd missed this whole thread. Repbot integration is THE major omission in JavaFibs IMO and your script will help until Cthulu approves the new release inim is busy on.

Steve

[Patti]

I do not believe that this is an acceptable use of the savedgames information-- you're using the savedgames information to mean something significantly different than the way that it is defined.  The savedgames list is *not* a list of dropped or abandoned (or even intentionally saved) matches, and should not be treated as though it is.

Having a large number of saved games does not make one a villain.  In fact, it may well mean that someone has recently played a significant number of matches to completion.

[wintom]

Having a large number of saved games does not make one a villain.  In fact, it may well mean that someone has recently played a significant number of matches to completion.

That's true and the opposite is also quite often the case: Somebody with less than ten saved games might still be a dropper who recently returned from his summer vacation.  The saved games list is of no use if it is not accessible without delay. That's why I wrote this script. In combination with RepBot there is a good chance to avoid droppers and also to recognize the very few false positives (mostly bots) given by the saved games list.
A similar script written in Perl exists since 2005 and hasn't done any harm. It's the users job to set the threshold to an acceptable value and to always keep in mind that even somebody with 20 saved games doesn't have to be a dropper. My advice is to crosscheck with RepBot.
Though I even play unlimited matches on Fibs from time to time, I cannot remember to have ever reached more than 10 saved games. It shouldn't be that difficult to keep that number low.
If there was a better solution I'd go for it, but there is not!

Thomas

[Patti]

A better solution would be to use the information in an advisory manner, rather than adopting a scorched-earth policy with data that don't support such a drastic response.  If people start wildly and widely misusing the savedgames data, I'll probably pull the information down.

The savedgames list is *not* a droppers list.  It should not be treated as one.

[wintom]

A better solution would be to use the information in an advisory manner, rather than adopting a scorched-earth policy with data that don't support such a drastic response. If people start wildly and widely misusing the savedgames data, I'll probably pull the information down.

I guess the drastic response is on your side, but please don't hesitate to do what your intuition tells you.

Thomas

[inim]

As for winton's little hack, I consider it perfectly fine and legal usage of a public resource. Whoever runs that script gives a threshold for dropper definition, which is a perfectly valid and individual decision. My personal policy for accepting invites is similar, I won't accept invites from people with more than 15 saved games unless I know them. I use repbot score as a secondary indicator in border cases, but the thing I rely on mostly is # of saved games. Of course the definition is not as precise as it should be, due to technical limitations of the FIBS data provided in savedgames. Well beyond fixing, but there is nothing else to base decisions for accepting invites on.

The functionality provided by winton's class could be easily integrated into JavaFIBS, and personally I have privately used, unpublished  code which uses the savedgames list in ways far beyond winton's script. Please note that pulling the savedgame list doesn't prevent one from creating a list of all fibs users even more detailed than savedgames. There is at least one other way to query this data from fibs.com I'm aware of. But that way puts a lot more load on fibs, and is harder to code. However, that algorithm can't produce the very data savedgame offers, namley missing "show saved USERID" command.


General thoughts on public data and access control:
--------------------------------------------------
I am a member of the Amazon Developer Network, and they have had a similar debate some years ago. Amazon offers it's extensive lists and product data publicly, and this was used by numerous "media database" tools. In short, pirates wrote software which augments your MP3 collection with images and information pulled from amazon.com. Needless to say amazon wasn't all happy with it. However, pulling the data was not considered an alternative, as their very business model approach aims at open and free information, and integration of it into 3rd party applications. The problem was solved with developer tokens. To access data, you by now have to give a combination of developer token and password in every URL that queries machine readable (read: XML) product data. Tokens are free, all you need to do is to enter your email into a form to get one.

So I suggest to simply use fibs username and fibs password to establish access control, if Patti want's to have some sort of access log. That data is there anyway, and writing the CGI script to enforce it is outright trivial.

JavaFIBS integration
--------------------

I have all the code needed to offer a solution similar to winton's script officially in JavaFIBS, however I am disturbed by Patti's threat to pull what I considered to be an official FIBS interface. I won't release software based on interfaces which are not official, however. So I'm definitely interested in this debate, with the goal to reach some form of officially supported interface status for the saved game list (and oter REST style webservices fibs.com de-facto offers, but let's delay that debate for now). The code itself is trivial, reading savedgames and creating a Map<PlayerID, List<PlayerID>> from it is about 20 lines in Java 5.

Patti, please tell me what form of integration would be OK in your book.

1) Displaying # saved games in the Player table
2) Offering automated update of villains lists based on a given saved games threshold
3) Offering automated update of villains lists based on a given saved games threshold in combination with other criteria, such as repbot score
1-3') The above functionality, but based on an average created from cached savedgames, see burper's comment

Personally, I root for option 3' and for myself use that criterions. But I won't release any code or binary Patti opposes.

Minimizing load and improving reliability of savedgames data
----------------------------------------------------------
First of all, I support burper's view as expressed here: http://www.fibsboard.com/javafibs/javafibs-2001-v1010-is-now-available-for-download-t1850.0.html;msg13962#msg13962

As burper suggested, caching savedgame and creating averages over it improves reliability. JF already has a workspace of persistent memory, so keeping average figures is trivial to implement in there. The other thing would be load on the fibs.com HTTP server, which will surely increase if JF or other clients automatically download the savedgame list.  JF would enforce a "once per day" policy to keep the load low.

So, Patti: is it acceptable to offer an option in the official JF release which would automatically downloads savedgames and offers end user decisions based on that data?

[wintom]

Thanks inim for this detailed analysis. I guess everybody's policy for accepting invites is based on saved games and RepBot as you stated due to the fact that there is no other variable to consider. That's why Patti's point of view doesn't sound too convincing at least to me. The villains list is a supportive security measure ecpecially for times where RepBot is asleep.

Anyway: Patti, if you want me to remove the script in order to save the saved games list, I will do so. But please give inims and my explanations another thought. We all know you are at the controls.

Thomas

[Hardy_whv]

I think in my 5 years on FIBS I never had more than 2 or 3 unfinished matches. What cases might there be for a player to have 15 or 20 unfinished matches? I always avoid players with such a list of unfinished matches. What other information should one utilize to avoid droppers, Patti? You don't like RepBot, you don't agree with the way of wintoms use of the SavedGames list. What do you recommend to successfully avoid droppers? One way would be to only play players that one knows personally. But this would it make impossible for new FIBSters to find opponents to play with. Just cusious.

As long as there is no automatic way to punish droppers on FIBS (i.e. adjusting rating depeding on the winning chances of both players, after a match was dropped), there should be information available that allows players to avoid droppers.

Hardy 

[stog]

hi all
ive been using the script for a few weeks now - set to a threashold of 25 saved games -- it gives quite a list still. i seem to have had no dropped games --- but i have had to leave many and pick up later because of javafibs and my mac's network config/firewall and router; so i might appear as a dropper to some with a script and a low threashold.

thx for all your efforts

[Patti]

The savedgames list is not and was never intended to be an official FIBS resource.  It's a quick hack that I pulled together one evening on an experimental basis, and while it's continued to run for quite a while now I don't particularly commit to supporting it.  If something were to happen tonight to cause the process to fail, it might not get fixed for days, weeks, or forever.

My understanding is that the villains list is (a) essentially permanent, and (b) drastic.  If I'm understanding correctly, it keeps the person from even talking to you.  I can assure you that perfectly innocent people will occasionally wind up with high savedgames counts, and the idea that someone might block them forever because of that really feels wrong with me.

In particular, if my understanding is correct it means that someone who is blocked because of the savedgames list can't even communicate with the blocker to resolve the situation.  If this is the case, I can assure you that such a problem will land in my lap more than once in the future.  As you know, I'm strongly motivated to avoid doing technical support for things that have nothing to do with FIBS itself.


You mentioned RepBot integration.  Bear in mind that RepBot is also not an official part of FIBS.  I consider it to be fundamentally broken and a royal pain in my ass, and every single time I get a complaint about it I'm sorely tempted to fire up my script that kills user accounts and then takes over the username.  "I have a bad reputation because A, B, and C complained about me but I never played any of them.   Fix this for me."  Sometimes they say please, but often not. 

Another one is, "It says I have 11 savedgames but I really only have six.  Fix this!"  The savedgames page has some fairly strong caveats on it about what the data do and don't mean, and why it's not what they think it is.  One of the issues I have with consuming and redisplaying the savedgames information is that those warnings seem to get lost, and the data are therefore interpreted as meaning something entirely different from their definition.  I don't really want to see those numbers displayed without a warning that they include unlimited matches and recently-completed matches.  (This is another beef that I have with RepBot.)

(In the past, I've come up with a relatively simple change to RepBot that would turn it into a much more accurate resource, and one that I would despise a great deal less.  If that ever gets implemented, I may reconsider my stance.)


Someone else asked why someone might legitimately have many savedgames.  Some people have savedgames because they got dropped.  Some because they intentionally postponed some games for later.  A few people play either very-long or unlimited matches with friends over extended periods of time... they'll play half an hour tonight, an hour next week, etc.  There are plenty of legitimate reasons to have saved games that don't involve intentionally dropping.

As for how to avoid droppers, it's not that hard.  I haven't been dropped in years, either on my Patti account or on my super secret nobody-knows-it's-me account.  I never play anyone with an low experience unless they've entered an email address.  I also never play anyone whose rating is vastly out of line with their experience.  Someone has 1800 experience after 58 games?  I don't expect that I have any chance of winning that match.

[Tom]

(Is this a JF thread or RebBot?  )

From my reading of the Repbot board there is a lot more useful information given from the FRIENDS command

I agree any information from repbot (or other automatic source) should not be used to totally Block Communications between players

I feel a correct repbot integration should be to display info from repbot on the invite box and allow the user to decide whom to accept from.

I also think that some repbot info could be displayed when you INVITE someone not on your friends list.

Yes this would add an extra step (optional?) but is well worth it.

tom

[spielberg]

(In the past, I've come up with a relatively simple change to RepBot that would turn it into a much more accurate resource, and one that I would despise a great deal less.  If that ever gets implemented, I may reconsider my stance.)

Details of this please Patti - you tend to only have good ideas!



Steve

[Patti]

Easy, spielberg.  In my mind, the number one issue with RepBot is that you can vouch for or (more commonly) complain about anyone.  This leads to situations where somebody pisses off one person, and that person either enlists their friends to gang up on the victim, or enlists their pseudo-friends (i.e. sock puppet accounts) to do the same.  I've deleted hundreds and hundreds of bogus sock puppet accounts that were created and had their experience pumped just for this purpose.

The change is this:  you can only complain about or vouch for someone within a fixed period of time (say one day) after you start, finish, or resume a match with them.  It would require RepBot to monitor the "toggle report" output and maintain some state data internally, but that's not terribly difficult to do.

This would make RepBot's information far more accurate, and significantly reduce the amount of tech support email I get about the bloody thing.  Without it, it's constantly hovering on the edge of getting the boot.

[webrunner]

Patti, that is a brilliant idea and should be implemented!!!

[inim]

Before we drift further away from the simple original point of this thread: it was about reading the "savedgames" list an presenting the figures to the user in some form, either standalone, averaged over a period of time, or in combination with other pieces of information such as repbot ratings.

You probably can end any debate by making it complicated enough and by moving the subject far enough from the original question. However, I still haven't seen the real question answered.

The code itself is trivial, reading savedgames and creating a Map<PlayerID, List<PlayerID>> from it is about 20 lines in Java 5.

Patti, please tell me what form of integration would be OK in your book.

1) Displaying # saved games in the Player table
2) Offering automated update of villains lists based on a given saved games threshold
3) Offering automated update of villains lists based on a given saved games threshold in combination with other criteria, such as repbot score
1-3') The above functionality, but based on an average created from cached savedgames, see burper's comment

Improving Repbot is a completely different and unrelated cup of tea, as 4 in 6 of the above options do not even involve it.

[inim]

Here are my 0.02 EUR / 0.03 USD on FIBS in the age of service oriented architectures, and the state of client side technology.

Fibs is rather small data collection, at the time of this writing is stores data for 6246 users. Recent runtime environments, such as Java, dot-net, and even scripting frameworks such as Perl or Python, come bundled or are easily bundled with databases in even the end user editions.

Speaking for Java, this would be JDK 6 + JPA (Java Persistency API) + Hibernate Entity Manager -- yielding a free (as in beer and in freedom) relational, fully transactional RDBMS in pure Java with a standardized object-relational persistence layer. 10 years ago this was high end technology and research, today it is installed on any JDK 6 enabled end user PC.

Here is what this enables for FIBS apps. Note I don't propose to do it all, and not without agreement of involved parties. However, coding this is very feasible today.

1) The database: The idea is simply to maintain a complete user database of all FIBS users in every local JavaFIBS. Technically this is a file of a few MB, maybe some 10 MB - by any standard a joke these days. I just bought a 750 GB harddisk yesterday for 140 bucks. Using the same code, bots can be created just as easily.

2) Bootstrap and recording CLIP: The User DB now can be easily initialized by running the algorithm yielding all current FIBS users. This is a sequence of a few dozen FIBS commands, around 70. From this bootstrap data, all game events from CLIP can be assigned as to users  as long as the JF user is logged in, or always if the same code runs inside a bot. I would think that we won't go for a full data mining set, but e.g. the own game history including opponent and date would be helpful. For all other users the date and opponent history for the last n (maybe 10) games would be sufficient. I'm sure the FIBS users will present their wishlist here anyway. Please note that in reqular intervals, say weekly, the bootstrap can be repeated to purge the DB from expired users.

3) Integrating 3rd party data: The next step is to integrate 3rd party sources from the internet, holding additional information for users. That would be FIBS savedgames (Map<User, List<User>>). Repbot score(Map<User, Double>), Repbot relation graph (user -> 2 lists of all vouchers and complaints for a user). Then there is fibsboard, which has Images, Address data, RSS feed, and a lot more. All data available from those bulk sources would be updated periodically by simply transfering XML data (dumps such as savedgames or the repbot mysql dump) provided by the operator of the external source. It COULD be done even against the will of the operator, but that is a messy and slippery slope, so at least JF never will offer those type of interfaces (aka wrappers).

4) User interaction: There is nothing in the architecture which prevented end users to intertact, i.e. insert, update, delete and query data. E.g. TD/Tourney info, written notes, etc. pp. The technology for this would be either direct access to the database via JDBC (using SQL), Java's own object-relational query language, or an easily set up Scripting component bundled with each JDK 6 and up. It allows persistent objects to be manipulated in Python, JavaScript, Groovy, and other languages. Given this facilities, the original issue of filtering user data by # of saved games becomes pretty much trivial - and such filters can be pretty complex scripts still maintainable by end users.

5) On imperfect sources: In this architecture, every piece of information is a tuple (datum, timestamp-of-retrival) anyway.  Outdated data is just what ALL webservices operate on, and there are well understood ways to deal with that. I don't think anybody here would say Amazon sucks just because the product data they just sent you in XML was cached and is outdated by 12h ... FIBS never was a transactional system, and no matter how hard we try will never become one. But this should not be standing in the way of progress.

Long story short, you end up with a fairly rich database for each fibs user, all stored locally on your disk. A plentora of new end user features becomes feasible with that. But all this requires to drop an outdated belief: FIBS is _not_ only what you can see after a login and using commands. Fibs has become a patchwork of services, some using CLIP, some using tell, and some using HTTP as a transport vehicle. Once you accept this truth, most other points follow naturally. Also "the bot" is no longer the only as a vehicle to extend FIBS -- we can do much of it on the client side today.

P.S. Opinions expressed here are mine and not necessarily shared by other JF developers.

[adrian]

I would love to see  at least 1) and 2) officially implemented but I doubt "the boss" will agree. Nevertheless, it is probably done already by a few users, since CLIP messages are "free" and may be read by anyone who logs in.

[inim]

Just learned that 3DFibs happily downloads "savedgames" ...

I can't say whether or not it displays the info and in what way, because it keeps crashing (maybe because my security subsystem and it's adware-components don't get along too well ?!) ... Could somebody with a less paranoid system kindly provide that info?

[FrankBerger]

Speaking for Java, this would be JDK 6 + JPA (Java Persistency API) + Hibernate Entity Manager -- yielding a free (as in beer and in freedom) relational, fully transactional RDBMS in pure Java with a standardized object-relational persistence layer. 10 years ago this was high end technology and research, today it is installed on any JDK 6 enabled end user PC.

1) The database: The idea is simply to maintain a complete user database of all FIBS users in every local JavaFIBS.


4) It allows persistent objects to be manipulated in Python, JavaScript, Groovy, and other languages. Given this facilities, the original issue of filtering user data by # of saved games becomes pretty much trivial - and such filters can be pretty complex scripts still maintainable by end users.

just my 2 pennies: First an integrated DB is a very good idea
- You are right that the JDK includes a DB, but that request the user to have a JDK. I guess only a fraction has it, so one has to inlude it in the download. The question is which DB to include? JavaDB (aka Apache Derby) or another, namely H2 (http://www.h2database.com/)  I'm currently testing both and I'm nearly sure that H2 is  a better choice for a single user system in every respect. It's  extremely fast, better handable (less files), just 1MB instead of 2MB, better recovery after a crash (do you want 2 explain users how to remove db.lck? but I have still to verify that). I can tell you some more in a few days.

- the programming API. I question whether JPA+Hibernate is the way to go. JPA is a pretty good API (much better than I ever expected, giving that people with very different interests made it) but the reference implementation (based on Toplink) is already 2 MB. I don't know Hibernate, but the last time I looked at it (in 2005) it needed a gazillion of other jars so the download will be enlarged by much and it had a pretty long startup time (it's fokus seems 2b appserver where startup time is negelectable). I would say if JPA, then stick with toplink-essential. I thought about such things for BGBlitz and decided to use iBatis (which has less than 400KB) what takes care for a lot of boring JDBC stuff but for some reason I had to change more than I wanted, so at the end of the day I polished and cleaned up my JDBC code.   
Depending on the amount of things and tables, JDBC is a viable option. It is not cool and more work, but keeps the download slim.

- I think if we offer to the end user SQL that will be fine already. I question whether it is worth to include a scripting language to satisfy the needs of the copule of users that are educated enough to use it, but not educated enough to use SQL. It's cool to offer an API for a scripting language, but is it a burning demand? And it enlarges the download again for a few MB (groovy: 2.7 MB). In the times of DSL that is nothing, but I think DSL is not everywhere.

so in 2 sentences:
- use H2DB/Toplink Essential or plain JDBC/scripting languages if there is a demand

ciao
Frank

[webrunner]

  thanks for your feedback Frank!!

[Chris]

To keep it short: I don't like the idea of having all fibs users a big brother watching you.

I know that it's easy programmiing your private proxy but still it is something different.

[don]

What I see here is a JavaFIBS abuse of RepBot which Patti objects to, and a suggestion to modify RepBot by Patti which makes sense.

Does that sum it up?

--
don

[Tom]

Database synchronization in systems with a well defined dbms is enough of a problem, let alone trying to collect data from bots

I think RepBot is meant to used as a GUIDE for players to assist them in deciding to invite/accept a match with a specific player.

That information can be gotten in real-time very easily, I think that an external data base is over-kill.

tom

[inim]

This isn't a repbot thread and never was one, sigh. Could we just leave this dead end issue called "repbot" and go back to the subject now? It was: is using the savedgames list considered proper behaviour for FIBS clients.

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg13779#msg13779

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg14001#msg14001

Hmmmm ... Pawlow. Pawlow. It doesn't ring a bell for me.

[inim]

Database synchronization in systems with a well defined dbms is enough of a problem, let alone trying to collect data from bots

No, it is not a problem we have. There are perfectly well deveolped methods and theory for working with timestamped data, and the rich SOA architectures and webservices we have today show they work perfectly well. It only becomes a problem if you try to built transactional systems based on timestamped data, but nobody suggested to do that.

As for collecting data: I don't consider "wget http://www.fibs.com/savedgames/list.html" to be overly complicated ...

That information can be gotten in real-time very easily, I think that an external data base is over-kill.

We can get it in realtime, true. But saved matches (i.e. for the Stats dialog) and other things need a persistent state. JF currently maintain it's persistent status in the form of 6 or 7 files. This is worse then using a database. My use of "database" above was as in "fine grained, persistent data storage", not as in "fully fledged transactional RDBMS".

I'd go with any persistent storage better than files, but those based on relational technology just happen to be the standardized way to do it in Java. You won't notice it is a RDBMS storage at all, as the JPA ORM (Object-Relational-Mapper) hides that implementation detail for you anyway. All you see is persistent Objects in JPA, unless you WANT to see the tables -- which is not needed. Given the state of the art, using JPA gives us LESS code as compared to the current file based persistency, along with better performance.

The price to pay will be 3rd party libs, totaling about 3-6 MB and a dependency on Java 5. That's it. Please also note the database will not be "external", it is just a library used by the JF binary. No end user will ever see any database process running, nor does he have to install anything.

[Tom]

This isn't a repbot thread and never was one, sigh. Could we just leave this dead end issue called "repbot" and go back to the subject now? It was: is using the savedgames list considered proper behaviour for FIBS clients.

So sorry... look Up There ^ by the word "Re:" I must have misunderstood, I thought *that* was the Subject Line

All I can see is reams of data for all of fibs flying around and there are only about 5-10 players I play on a given day.

Sounds like needless bandwidth

SOA is totally different, there is an urgent need to have that db spread out

Apples and Oranges

Besides I do not recall any time stamps on data from bots... time stamps are useless if they are not from
an authoritative source...

tom

[inim]

Besides I do not recall any time stamps on data from bots... time stamps are useless if they are not from
an authoritative source...

$>wget -d http://www.fibs.com/savedgames/list.html
DEBUG output created by Wget 1.10.2 on cygwin.

[...]
GET /savedgames/list.html HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: www.fibs.com
Connection: Keep-Alive

[...]
HTTP/1.1 200 OK
Date: Sat, 15 Mar 2008 12:45:03 GMT
Server: Apache/1.3.20 (Unix)  (Red-Hat/Linux) DAV/1.0.2 PHP/4.0.6 mod_perl/1.24_01
Last-Modified: Sat, 15 Mar 2008 10:40:03 GMT
ETag: "44805b-47a72-47dba783"
Accept-Ranges: bytes
Content-Length: 293490
Connection: close
Content-Type: text/html

Good enough?

All I can see is reams of data for all of fibs flying around and there are only about 5-10 players I play on a given day.
Sounds like needless bandwidth

293490 Mar 15 11:40 list.html
  88713 Mar 15 11:40 list.packed.html.gz

Simply adding gz to the cron-job creating the list makes it an 88K download. There should be bandwidth enough to download 88k per user, given the benefit that data has.

So sorry... look Up There ^ by the word "Re:" I must have misunderstood, I thought *that* was the Subject Line

The subject line is a misnomer if you look at the actual content of both the original post and the following debate. I'd ask you as an Admin of this forum to correct it to "SavedGames".

[Tom]

With this thread being as long as it is and some of the posts being rather long winded...
(I honestly think there is a good idea getting lost in the shuffle)

in about 25 words, what would you use the saved games list for?

and in another 25 words (give or take) how often would sync the 88k data? (wget is all or nothing, isn't it?)
(Is that file updated once a day? I see it has not changed in the last 2 hours...)

And all things being equal can you imagine a scenario where I would consume 88k of interaction with RepBot during the normal course of play?

Might it be better to take this application to the website? maybe a data miner that post more useful pages to fibs.com ?

tom

[inim]

With this thread being as long as it is and some of the posts being rather long winded...
(I honestly think there is a good idea getting lost in the shuffle)

The idea is ages old, and implemented by other apps for long (3DFibs, eg.). This is a an attempt of a serious software-engineering debate on if and how to realize it in JF. And I really wished you (and other posters) would actually read the design sugestions and comment on them, content wise -- rather than emitting <stdin.h> type comments triggered by Pawlow reflexes to buzzwords like "repbot" or "database". Sorry, but I'm a little *bleeep*ed here. Somehow my impression is that the interest in serious improvements to FIBS and the required engineering to get there is of as much interest to FIBSters as the latest car model is to the Amish. I only have so and so much energy, and when it runs out i happily leave you with your horse carts.



Ok, friends again. As for your questions:

in about 25 words, what would you use the saved games list for?

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg13968#msg13968
Section "JavaFIBS integration"

and in another 25 words (give or take) how often would sync the 88k data?

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg13968#msg13968
Section "Minimizing load and improving reliability of savedgames data"

Might it be better to take this application to the website?

No.

[Tom]

Well, I took a second look and this comment should kinda end the discussion...
why implement a feature on something that is not really a supported feature?

The savedgames list is not and was never intended to be an official FIBS resource.  It's a quick hack that I pulled together one evening on an experimental basis, and while it's continued to run for quite a while now I don't particularly commit to supporting it.  If something were to happen tonight to cause the process to fail, it might not get fixed for days, weeks, or forever.

Also when you can get possibly more meaningful information from supported sources, such as repbot

And if you want an engineering debate, why ignore this question?

And all things being equal can you imagine a scenario where I would consume 88k of interaction with RepBot during the normal course of play?

It deals with raw bandwidth and resources used on the system, which is not a factor that can be ignored.

As far as providing useful, up to date information to the player I rather like the way DelFIBS handles it, it asks repbot in real-time...

tom

[inim]

Well, I took a second look and this comment should kinda end the discussion...
why implement a feature on something that is not really a supported feature?

Great, they just outlawed the wheel so let's all just trash our cars. What about the fact that Patti is dead wrong in her decision? I'm trying to find a technical solution which minimizes work for her, respects her concern w/r to timestamped and inherently less-than-perfect data, but still enables progress.

Why shall all client authors write hundreds if not thousands of lines of work-around code just because Patti denies to write 20 in a server side CGI? I just don't get it. And in case you missed it, it takes two for FIBS to work, the clients and the server. The client author's problems are as real and valid as Patti's.

This thread is all about finding a technical solution which respects both, the requirements client authors have and concerns Patti has. The vehicle for this is the savedgames list. That's it in a nutshell.

Also when you can get possibly more meaningful information from supported sources, such as repbot

Repot is in no way supported, Patti threatens to ban it about 3 times a year, and Avik isn't actively developing it anymore. And we don't need no stinking Repbot when we can get the same information in a much easier and less resource consuming way from savedgames. Repbot just uses the data from that very same source itself, btw.

It deals with raw bandwidth and resources used on the system, which is not a factor that can be ignored.

The traffic caused by Repbot queries is much more resource savy than a simple http download of a tiny file. And repbot is just as well supported and official as the savedgames list is. I won't even mention that a savedgames based solution requires mabye 20 lines of code, Repbot parsing maybe 1000. I'm just the idiot who has to write that code, so why the hack I care. So sorry I am looking looking for a PERMANENT and CLEAN solution to a common problem. See here:

http://www.fibsboard.com/3dfibs/invitation-filter-suggestion-t1740.0.html

BTW, I didn't ignore your question, you ignored my proposal which had a section on the issue before you even asked.

As far as providing useful, up to date information to the player I rather like the way DelFIBS handles it, it asks repbot in real-time...

Between there and JF now are ONLY a few thousand lines of code and a new Parser architecture. We simply can't do it like Delfibs easily because our code base doesn't allow it. We CAN however easily add info we would get from savedgames, as this technically is a way easier task. Hint: HTTP dowloads don't have to go through the CLIP Parser, but Repbot replies do.

Thus: let's get back to work. We need to design a solution Patti can support and from which all clients will benefit. Simply let's do baby steps and engineer a better FIBS. That's at least what I try. And if you look back to the very first posting, I think it's just what winton tried as well.

[burper]

How about JF sharing information automatically with other JF's?
Use chat messages with some special preface character for this C2C communication.

What to share:
-friends/villians list
-playing history, i.e. who has dropped you, who you have played a lot

Who to share it with:
-those on your friends list
-those whom you play with "a lot"
-everyone except those on your villian list.
-not someone who's JF is not sharing with yours.
-not someone you have a saved game with

Have the default be "no sharing", and let users configure what they want to share and
who they want to share it with.

What to do with the shared data:
keep it as your own personal repbot style database, except that you can configure how to interpret
the data, i.e. what constitutes a vouch or complaint, weighting,  and my favorite: et cetera.

The data could be shared in a compressed form, and only include changes since last update,
and potentially use IM instead of fibs.

[inim]

How about JF sharing information automatically with other JF's?
Use chat messages with some special preface character for this C2C communication.

Yes, I had that thought myself, but i wouldn't use a fixed char to bootstrap a session. Feel free to open a new thread for it, to which I happily will contribute more details. Please do not complicate this one, which is hard enough to keep on track already .

Beside C2C I have more ideas which would be useful for all clients (that support them), such as UTF-7O support and an improved form of the PPP (PlayerProfileProtocol) based on that C2C idea, mostly along the line of your proposal. But as said: babysteps. I'll be happy for now when we get the savedgames list operable. Let's just stay focused, ok?

Just expanding the idea of FIBS offering REST style SOA/Webservice interfaces is enourmously rich in itself, consider:

* Repository of known bots
* List of recently deleted players
* List of recently added players
* ...

So far only Patti knows what useful information can be obtained from the FIBS files with simple scripts. But I don't think the savedgames list is the end of the tunnel already. And given the effort it takes do server side CGI scripting vs. the benefits to clients, this concept has an enourmous leverage. REST services are the single most effective improvement of FIBS in terms of bangs per buck.

So let's try to convince Patti that helping client authors with server side services is A Good Thing™ for all, including herself. Once we did that, the next step is to get client authors talking about other cross client improvements, such as UTF-7O encoded communications or C2C.

P.S. I just had off-board communication with Patti, who is traveling currently. Thus the debate is dormant (and any decision pending) until she returns and can take part again.

[Patti]

* Repository of known bots

Not available in any reliable form.  Even I don't have those data, since a bot is just a normal FIBS user.  I try to keep a list so that I can filter them out of savedgames, but it's a manual process and quite hit-or-miss.  (I have a hard enough time just getting people to provide identity information for bots.)

* List of recently deleted players

Not available.

* List of recently added players

Not readily available, though it can be parsed out of the "toggle notify" information.

A lot of what you wish for isn't available, or is potentially available but would require some complex harvesting and storage algorithms on the server side.  FIBS is just not designed for stuff like this, and what seems like it should be simple is often quite the can of worms.  It's really hard to build a Ferrari with a stone chisel.

It seems to me that if you want REST services, you might be best served by setting up a bot that harvests visible data and provides it in the format you want.  That way you have complete control over what data you serve and how you serve them.

[webrunner]

That was exactly the idea what i had.
But don't you think that will make people more scared to talk freely on Fibs?

[inim]

OK, before we lose ourselves in another side-debate. Would it be possible to get a straight answer to original question, which was "can we make savedgames a supported interface to fibs data not available elsewhere but frequently requested by users of clients". Thx.

See:
http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg14001#msg14001

If the answer is "yes" in any form, we should continue and engineer some syntax. My suggestion is to create

1) An XML Schema which provides information about
- UTC time of instance creation
- UTC time of the next update of the instance
- Map<Player, List<Player>> in some XML serialization syntax.
2) Instances
of beforementioned Schema which are updated by a cron job running on fibs com and provided under a fixed URL.

I will happily provide the Schema in a machine readable form, and Perl code to produce instances. Next step would be to optimize to satisfy the constraints discussed above, such as bandwidth usage, displaying instances in human readable form, improving reliability, access control, update schedule etc. pp.

Please note that "yes, but it may go away anytime" isn't an satisfactory answer. I understand that due to problems in operations the service may be interrupted for some days, that's only normal. But what is needed to make it worth coding against such an API is a commitement to support it to the same degree fibs.com's other services are supported.

[Patti]

As I said earlier, I can't guarantee that the savedgames information will continue to be there in the future.  It was a quick hack that I pulled together in about 20 minutes one evening  just to see how well it would work, and it was never intended to be an official FIBS resource.  I don't monitor the process that generates it, and while it's been quite stable thus far I probably wouldn't notice for a while if it went south.

The more I see decisions being made based on misunderstanding of the information, the less likely I am to want to continue providing it.  Just today, I had the joy of witnessing something like this:

(someone) shouts:  Don't play so-and-so, he's a dropper.
(someone else) shouts: He is?
(someone) shouts: Yeah, he has 20 saved games.  He's a dropper.

I was curious, so I went off and reviewed the guy's saved games.  In the vast majority of them he was either significantly in the lead, or the game was about even... far from a classic dropper pattern.

Beyond that, I'm coming to understand that the numbers are a less accurate predictor than I hoped they'd be.  A couple of weeks ago I investigated someone in the dropper's hall of shame, only to find out that most of his matches were actually finished.  He'd played quite a few matches in the hours before the list was calculated, and that was enough to push him into the top 30 list.

In short, I'm not too enthusiastic about providing an improved interface to data that have proven to be unreliable, that are often misinterpreted, and that might cause unintended harm to innocent people.

(Oh, and if I *was* going to provide an official API, it wouldn't be XML.  Morse code would be more likely than XML, or maybe Swahili and BCD... I despise XML.  But that's another topic altogether.)

[spielberg]

This is off topic but a point of information to clarify Patti's last statement:

I'm far less discreet than her Pattiness and have realised for some time that the "Hall of Shame" contains at least one non dropper; in this case the fibster Patti is referring to is rif_raf who's not only a nice guy but also plays bg as (IMO) it should be played with plenty of side chat. Backgammon is a social game, if I wanted silence I'd play chess. Be warned tho' rif's game is better than his rating --- he's learning quite quickly.


Steve

[Patti]

Actually, spielberg, it wasn't rif_raf, but someone else.  Their appearance on the hall of shame page was a temporary aberration-- they were only there for one day.  I'd hate for people to shun someone just for something like that.