RepBot/SavedGames

[inim]

Here are my 0.02 EUR / 0.03 USD on FIBS in the age of service oriented architectures, and the state of client side technology.

Fibs is rather small data collection, at the time of this writing is stores data for 6246 users. Recent runtime environments, such as Java, dot-net, and even scripting frameworks such as Perl or Python, come bundled or are easily bundled with databases in even the end user editions.

Speaking for Java, this would be JDK 6 + JPA (Java Persistency API) + Hibernate Entity Manager -- yielding a free (as in beer and in freedom) relational, fully transactional RDBMS in pure Java with a standardized object-relational persistence layer. 10 years ago this was high end technology and research, today it is installed on any JDK 6 enabled end user PC.

Here is what this enables for FIBS apps. Note I don't propose to do it all, and not without agreement of involved parties. However, coding this is very feasible today.

1) The database: The idea is simply to maintain a complete user database of all FIBS users in every local JavaFIBS. Technically this is a file of a few MB, maybe some 10 MB - by any standard a joke these days. I just bought a 750 GB harddisk yesterday for 140 bucks. Using the same code, bots can be created just as easily.

2) Bootstrap and recording CLIP: The User DB now can be easily initialized by running the algorithm yielding all current FIBS users. This is a sequence of a few dozen FIBS commands, around 70. From this bootstrap data, all game events from CLIP can be assigned as to users  as long as the JF user is logged in, or always if the same code runs inside a bot. I would think that we won't go for a full data mining set, but e.g. the own game history including opponent and date would be helpful. For all other users the date and opponent history for the last n (maybe 10) games would be sufficient. I'm sure the FIBS users will present their wishlist here anyway. Please note that in reqular intervals, say weekly, the bootstrap can be repeated to purge the DB from expired users.

3) Integrating 3rd party data: The next step is to integrate 3rd party sources from the internet, holding additional information for users. That would be FIBS savedgames (Map<User, List<User>>). Repbot score(Map<User, Double>), Repbot relation graph (user -> 2 lists of all vouchers and complaints for a user). Then there is fibsboard, which has Images, Address data, RSS feed, and a lot more. All data available from those bulk sources would be updated periodically by simply transfering XML data (dumps such as savedgames or the repbot mysql dump) provided by the operator of the external source. It COULD be done even against the will of the operator, but that is a messy and slippery slope, so at least JF never will offer those type of interfaces (aka wrappers).

4) User interaction: There is nothing in the architecture which prevented end users to intertact, i.e. insert, update, delete and query data. E.g. TD/Tourney info, written notes, etc. pp. The technology for this would be either direct access to the database via JDBC (using SQL), Java's own object-relational query language, or an easily set up Scripting component bundled with each JDK 6 and up. It allows persistent objects to be manipulated in Python, JavaScript, Groovy, and other languages. Given this facilities, the original issue of filtering user data by # of saved games becomes pretty much trivial - and such filters can be pretty complex scripts still maintainable by end users.

5) On imperfect sources: In this architecture, every piece of information is a tuple (datum, timestamp-of-retrival) anyway.  Outdated data is just what ALL webservices operate on, and there are well understood ways to deal with that. I don't think anybody here would say Amazon sucks just because the product data they just sent you in XML was cached and is outdated by 12h ... FIBS never was a transactional system, and no matter how hard we try will never become one. But this should not be standing in the way of progress.

Long story short, you end up with a fairly rich database for each fibs user, all stored locally on your disk. A plentora of new end user features becomes feasible with that. But all this requires to drop an outdated belief: FIBS is _not_ only what you can see after a login and using commands. Fibs has become a patchwork of services, some using CLIP, some using tell, and some using HTTP as a transport vehicle. Once you accept this truth, most other points follow naturally. Also "the bot" is no longer the only as a vehicle to extend FIBS -- we can do much of it on the client side today.

P.S. Opinions expressed here are mine and not necessarily shared by other JF developers.

[adrian]

I would love to see  at least 1) and 2) officially implemented but I doubt "the boss" will agree. Nevertheless, it is probably done already by a few users, since CLIP messages are "free" and may be read by anyone who logs in.

[inim]

Just learned that 3DFibs happily downloads "savedgames" ...

I can't say whether or not it displays the info and in what way, because it keeps crashing (maybe because my security subsystem and it's adware-components don't get along too well ?!) ... Could somebody with a less paranoid system kindly provide that info?

[FrankBerger]

Speaking for Java, this would be JDK 6 + JPA (Java Persistency API) + Hibernate Entity Manager -- yielding a free (as in beer and in freedom) relational, fully transactional RDBMS in pure Java with a standardized object-relational persistence layer. 10 years ago this was high end technology and research, today it is installed on any JDK 6 enabled end user PC.

1) The database: The idea is simply to maintain a complete user database of all FIBS users in every local JavaFIBS.


4) It allows persistent objects to be manipulated in Python, JavaScript, Groovy, and other languages. Given this facilities, the original issue of filtering user data by # of saved games becomes pretty much trivial - and such filters can be pretty complex scripts still maintainable by end users.

just my 2 pennies: First an integrated DB is a very good idea
- You are right that the JDK includes a DB, but that request the user to have a JDK. I guess only a fraction has it, so one has to inlude it in the download. The question is which DB to include? JavaDB (aka Apache Derby) or another, namely H2 (http://www.h2database.com/)  I'm currently testing both and I'm nearly sure that H2 is  a better choice for a single user system in every respect. It's  extremely fast, better handable (less files), just 1MB instead of 2MB, better recovery after a crash (do you want 2 explain users how to remove db.lck? but I have still to verify that). I can tell you some more in a few days.

- the programming API. I question whether JPA+Hibernate is the way to go. JPA is a pretty good API (much better than I ever expected, giving that people with very different interests made it) but the reference implementation (based on Toplink) is already 2 MB. I don't know Hibernate, but the last time I looked at it (in 2005) it needed a gazillion of other jars so the download will be enlarged by much and it had a pretty long startup time (it's fokus seems 2b appserver where startup time is negelectable). I would say if JPA, then stick with toplink-essential. I thought about such things for BGBlitz and decided to use iBatis (which has less than 400KB) what takes care for a lot of boring JDBC stuff but for some reason I had to change more than I wanted, so at the end of the day I polished and cleaned up my JDBC code.   
Depending on the amount of things and tables, JDBC is a viable option. It is not cool and more work, but keeps the download slim.

- I think if we offer to the end user SQL that will be fine already. I question whether it is worth to include a scripting language to satisfy the needs of the copule of users that are educated enough to use it, but not educated enough to use SQL. It's cool to offer an API for a scripting language, but is it a burning demand? And it enlarges the download again for a few MB (groovy: 2.7 MB). In the times of DSL that is nothing, but I think DSL is not everywhere.

so in 2 sentences:
- use H2DB/Toplink Essential or plain JDBC/scripting languages if there is a demand

ciao
Frank

[webrunner]

  thanks for your feedback Frank!!

[Chris]

To keep it short: I don't like the idea of having all fibs users a big brother watching you.

I know that it's easy programmiing your private proxy but still it is something different.

[don]

What I see here is a JavaFIBS abuse of RepBot which Patti objects to, and a suggestion to modify RepBot by Patti which makes sense.

Does that sum it up?

--
don

[Tom]

Database synchronization in systems with a well defined dbms is enough of a problem, let alone trying to collect data from bots

I think RepBot is meant to used as a GUIDE for players to assist them in deciding to invite/accept a match with a specific player.

That information can be gotten in real-time very easily, I think that an external data base is over-kill.

tom

[inim]

This isn't a repbot thread and never was one, sigh. Could we just leave this dead end issue called "repbot" and go back to the subject now? It was: is using the savedgames list considered proper behaviour for FIBS clients.

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg13779#msg13779

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg14001#msg14001

Hmmmm ... Pawlow. Pawlow. It doesn't ring a bell for me.

[inim]

Database synchronization in systems with a well defined dbms is enough of a problem, let alone trying to collect data from bots

No, it is not a problem we have. There are perfectly well deveolped methods and theory for working with timestamped data, and the rich SOA architectures and webservices we have today show they work perfectly well. It only becomes a problem if you try to built transactional systems based on timestamped data, but nobody suggested to do that.

As for collecting data: I don't consider "wget http://www.fibs.com/savedgames/list.html" to be overly complicated ...

That information can be gotten in real-time very easily, I think that an external data base is over-kill.

We can get it in realtime, true. But saved matches (i.e. for the Stats dialog) and other things need a persistent state. JF currently maintain it's persistent status in the form of 6 or 7 files. This is worse then using a database. My use of "database" above was as in "fine grained, persistent data storage", not as in "fully fledged transactional RDBMS".

I'd go with any persistent storage better than files, but those based on relational technology just happen to be the standardized way to do it in Java. You won't notice it is a RDBMS storage at all, as the JPA ORM (Object-Relational-Mapper) hides that implementation detail for you anyway. All you see is persistent Objects in JPA, unless you WANT to see the tables -- which is not needed. Given the state of the art, using JPA gives us LESS code as compared to the current file based persistency, along with better performance.

The price to pay will be 3rd party libs, totaling about 3-6 MB and a dependency on Java 5. That's it. Please also note the database will not be "external", it is just a library used by the JF binary. No end user will ever see any database process running, nor does he have to install anything.

[Tom]

This isn't a repbot thread and never was one, sigh. Could we just leave this dead end issue called "repbot" and go back to the subject now? It was: is using the savedgames list considered proper behaviour for FIBS clients.

So sorry... look Up There ^ by the word "Re:" I must have misunderstood, I thought *that* was the Subject Line

All I can see is reams of data for all of fibs flying around and there are only about 5-10 players I play on a given day.

Sounds like needless bandwidth

SOA is totally different, there is an urgent need to have that db spread out

Apples and Oranges

Besides I do not recall any time stamps on data from bots... time stamps are useless if they are not from
an authoritative source...

tom

[inim]

Besides I do not recall any time stamps on data from bots... time stamps are useless if they are not from
an authoritative source...

$>wget -d http://www.fibs.com/savedgames/list.html
DEBUG output created by Wget 1.10.2 on cygwin.

[...]
GET /savedgames/list.html HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: www.fibs.com
Connection: Keep-Alive

[...]
HTTP/1.1 200 OK
Date: Sat, 15 Mar 2008 12:45:03 GMT
Server: Apache/1.3.20 (Unix)  (Red-Hat/Linux) DAV/1.0.2 PHP/4.0.6 mod_perl/1.24_01
Last-Modified: Sat, 15 Mar 2008 10:40:03 GMT
ETag: "44805b-47a72-47dba783"
Accept-Ranges: bytes
Content-Length: 293490
Connection: close
Content-Type: text/html

Good enough?

All I can see is reams of data for all of fibs flying around and there are only about 5-10 players I play on a given day.
Sounds like needless bandwidth

293490 Mar 15 11:40 list.html
  88713 Mar 15 11:40 list.packed.html.gz

Simply adding gz to the cron-job creating the list makes it an 88K download. There should be bandwidth enough to download 88k per user, given the benefit that data has.

So sorry... look Up There ^ by the word "Re:" I must have misunderstood, I thought *that* was the Subject Line

The subject line is a misnomer if you look at the actual content of both the original post and the following debate. I'd ask you as an Admin of this forum to correct it to "SavedGames".

[Tom]

With this thread being as long as it is and some of the posts being rather long winded...
(I honestly think there is a good idea getting lost in the shuffle)

in about 25 words, what would you use the saved games list for?

and in another 25 words (give or take) how often would sync the 88k data? (wget is all or nothing, isn't it?)
(Is that file updated once a day? I see it has not changed in the last 2 hours...)

And all things being equal can you imagine a scenario where I would consume 88k of interaction with RepBot during the normal course of play?

Might it be better to take this application to the website? maybe a data miner that post more useful pages to fibs.com ?

tom

[inim]

With this thread being as long as it is and some of the posts being rather long winded...
(I honestly think there is a good idea getting lost in the shuffle)

The idea is ages old, and implemented by other apps for long (3DFibs, eg.). This is a an attempt of a serious software-engineering debate on if and how to realize it in JF. And I really wished you (and other posters) would actually read the design sugestions and comment on them, content wise -- rather than emitting <stdin.h> type comments triggered by Pawlow reflexes to buzzwords like "repbot" or "database". Sorry, but I'm a little *bleeep*ed here. Somehow my impression is that the interest in serious improvements to FIBS and the required engineering to get there is of as much interest to FIBSters as the latest car model is to the Amish. I only have so and so much energy, and when it runs out i happily leave you with your horse carts.



Ok, friends again. As for your questions:

in about 25 words, what would you use the saved games list for?

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg13968#msg13968
Section "JavaFIBS integration"

and in another 25 words (give or take) how often would sync the 88k data?

http://www.fibsboard.com/javafibs/repbotsavedgames-t1819.0.html;msg13968#msg13968
Section "Minimizing load and improving reliability of savedgames data"

Might it be better to take this application to the website?

No.

[Tom]

Well, I took a second look and this comment should kinda end the discussion...
why implement a feature on something that is not really a supported feature?

The savedgames list is not and was never intended to be an official FIBS resource.  It's a quick hack that I pulled together one evening on an experimental basis, and while it's continued to run for quite a while now I don't particularly commit to supporting it.  If something were to happen tonight to cause the process to fail, it might not get fixed for days, weeks, or forever.

Also when you can get possibly more meaningful information from supported sources, such as repbot

And if you want an engineering debate, why ignore this question?

And all things being equal can you imagine a scenario where I would consume 88k of interaction with RepBot during the normal course of play?

It deals with raw bandwidth and resources used on the system, which is not a factor that can be ignored.

As far as providing useful, up to date information to the player I rather like the way DelFIBS handles it, it asks repbot in real-time...

tom

[inim]

Well, I took a second look and this comment should kinda end the discussion...
why implement a feature on something that is not really a supported feature?

Great, they just outlawed the wheel so let's all just trash our cars. What about the fact that Patti is dead wrong in her decision? I'm trying to find a technical solution which minimizes work for her, respects her concern w/r to timestamped and inherently less-than-perfect data, but still enables progress.

Why shall all client authors write hundreds if not thousands of lines of work-around code just because Patti denies to write 20 in a server side CGI? I just don't get it. And in case you missed it, it takes two for FIBS to work, the clients and the server. The client author's problems are as real and valid as Patti's.

This thread is all about finding a technical solution which respects both, the requirements client authors have and concerns Patti has. The vehicle for this is the savedgames list. That's it in a nutshell.

Also when you can get possibly more meaningful information from supported sources, such as repbot

Repot is in no way supported, Patti threatens to ban it about 3 times a year, and Avik isn't actively developing it anymore. And we don't need no stinking Repbot when we can get the same information in a much easier and less resource consuming way from savedgames. Repbot just uses the data from that very same source itself, btw.

It deals with raw bandwidth and resources used on the system, which is not a factor that can be ignored.

The traffic caused by Repbot queries is much more resource savy than a simple http download of a tiny file. And repbot is just as well supported and official as the savedgames list is. I won't even mention that a savedgames based solution requires mabye 20 lines of code, Repbot parsing maybe 1000. I'm just the idiot who has to write that code, so why the hack I care. So sorry I am looking looking for a PERMANENT and CLEAN solution to a common problem. See here:

http://www.fibsboard.com/3dfibs/invitation-filter-suggestion-t1740.0.html

BTW, I didn't ignore your question, you ignored my proposal which had a section on the issue before you even asked.

As far as providing useful, up to date information to the player I rather like the way DelFIBS handles it, it asks repbot in real-time...

Between there and JF now are ONLY a few thousand lines of code and a new Parser architecture. We simply can't do it like Delfibs easily because our code base doesn't allow it. We CAN however easily add info we would get from savedgames, as this technically is a way easier task. Hint: HTTP dowloads don't have to go through the CLIP Parser, but Repbot replies do.

Thus: let's get back to work. We need to design a solution Patti can support and from which all clients will benefit. Simply let's do baby steps and engineer a better FIBS. That's at least what I try. And if you look back to the very first posting, I think it's just what winton tried as well.

[burper]

How about JF sharing information automatically with other JF's?
Use chat messages with some special preface character for this C2C communication.

What to share:
-friends/villians list
-playing history, i.e. who has dropped you, who you have played a lot

Who to share it with:
-those on your friends list
-those whom you play with "a lot"
-everyone except those on your villian list.
-not someone who's JF is not sharing with yours.
-not someone you have a saved game with

Have the default be "no sharing", and let users configure what they want to share and
who they want to share it with.

What to do with the shared data:
keep it as your own personal repbot style database, except that you can configure how to interpret
the data, i.e. what constitutes a vouch or complaint, weighting,  and my favorite: et cetera.

The data could be shared in a compressed form, and only include changes since last update,
and potentially use IM instead of fibs.

[inim]

How about JF sharing information automatically with other JF's?
Use chat messages with some special preface character for this C2C communication.

Yes, I had that thought myself, but i wouldn't use a fixed char to bootstrap a session. Feel free to open a new thread for it, to which I happily will contribute more details. Please do not complicate this one, which is hard enough to keep on track already .

Beside C2C I have more ideas which would be useful for all clients (that support them), such as UTF-7O support and an improved form of the PPP (PlayerProfileProtocol) based on that C2C idea, mostly along the line of your proposal. But as said: babysteps. I'll be happy for now when we get the savedgames list operable. Let's just stay focused, ok?

Just expanding the idea of FIBS offering REST style SOA/Webservice interfaces is enourmously rich in itself, consider:

* Repository of known bots
* List of recently deleted players
* List of recently added players
* ...

So far only Patti knows what useful information can be obtained from the FIBS files with simple scripts. But I don't think the savedgames list is the end of the tunnel already. And given the effort it takes do server side CGI scripting vs. the benefits to clients, this concept has an enourmous leverage. REST services are the single most effective improvement of FIBS in terms of bangs per buck.

So let's try to convince Patti that helping client authors with server side services is A Good Thing™ for all, including herself. Once we did that, the next step is to get client authors talking about other cross client improvements, such as UTF-7O encoded communications or C2C.

P.S. I just had off-board communication with Patti, who is traveling currently. Thus the debate is dormant (and any decision pending) until she returns and can take part again.

[Patti]

* Repository of known bots

Not available in any reliable form.  Even I don't have those data, since a bot is just a normal FIBS user.  I try to keep a list so that I can filter them out of savedgames, but it's a manual process and quite hit-or-miss.  (I have a hard enough time just getting people to provide identity information for bots.)

* List of recently deleted players

Not available.

* List of recently added players

Not readily available, though it can be parsed out of the "toggle notify" information.

A lot of what you wish for isn't available, or is potentially available but would require some complex harvesting and storage algorithms on the server side.  FIBS is just not designed for stuff like this, and what seems like it should be simple is often quite the can of worms.  It's really hard to build a Ferrari with a stone chisel.

It seems to me that if you want REST services, you might be best served by setting up a bot that harvests visible data and provides it in the format you want.  That way you have complete control over what data you serve and how you serve them.

[webrunner]

That was exactly the idea what i had.
But don't you think that will make people more scared to talk freely on Fibs?