News:

All new Fibsboard.com for 2011 --" your unofficial Forum for all fings FIBS."
New Membergroups, new boards including topical discussions, Campaigns, Culture and Gossip. Photo Galleries, Forum Backgammon vs Top Guest, and much much more.........faster and improved, with sas :)

Main Menu

How do 'luck' calculations work..

Started by diane, June 26, 2010, 01:50:48 AM

Previous topic - Next topic

sixty_something

it all seems clear to me :lol:

OK, when words fail, resort to numbers .. i will provide a few examples of what i have observed and what i am trying to say .. while these will be incomplete at best, perhaps they will be helpful in developing a better understanding .. if not of my thesis, of the way "luck" is calculated .. i may be completely wrong and if so will readily admit it, but i think i can illustrate what i am trying to say .. however, it will take a while to create them .. so, don't hold your breath

meanwhile,  :stupid:
A little inaccuracy sometimes saves tons of explanation. -- Unknown
e-mail me

pck

#41
Quote from: Zorba on July 01, 2010, 09:31:35 PM
pck's theoretical argument is correct and is actually pretty trivial from the definition of luck, for a statistician.

Yes, it's mathematically trivial because there is hardly any math involved! I was mainly concerned with giving an explanation of how we arrive at the confused notion that "skill begets luck". What I wrote in #30 is not a theoretical argument (for it can neither be verified nor falsified), but an attempt at a conceptual elucidation by looking at how "luck" and "skill" are used in everyday language and how these terms are related to their mathematical counterparts.

Quote from: Zorba on July 01, 2010, 09:31:35 PM
I'd suggest just accepting that most people's "intuitive" or "perceptive" concept of probabilities or statistics as in "observing luck" are notoriously bad, skewed, inconsistent, etc. and therefore, you need to use mathematics to get correct answers to your questions.

I'd like to add that by setting down a formal definition for, say, "luck", we define what counts as "correct". There is no a priori concept of "the correct definition of luck" for us to discover. But in order for (mathematical) answers to make sense, there must be conceptual harmony between them and the questions they are answers to. And while using math is good for quantification and precision, its use alone does not guarantee the aforementioned harmony.

pck

#42
Quote from: Zorba on July 01, 2010, 08:17:13 PM
XG uses (your luck minus opponent's luck) for luck, like Snowie does (and unlike GnuBG does).

As to your other question, most matches "need" a lot of luck to be decided, since the skill difference alone won't be enough. Using GWC, say 20% can be gained in one game by the skill difference between a good player and a beginner. Then the best player still needs 30% luck to win, or the worst player 70%.

If that amount of luck hasn't been reached and the game is nearing the end (no contact), then barring any super blunders in the endgame, a lot of luck has to happen for either player.

The opposite is not true, if one player already has been very lucky, he's likely to have really high winning chances, like 80% or more. Then either he also gets the remaining amount of luck to win the game, or the other player gets extreme luck in the end. An amount of luck that brings the total luck back to around zero like you describe, could not finish the match (unless there was an extreme skill difference), as it would bring both players back to some wide margin around 50% GWC (say, 30-70%).

Ok, you're referring to the fact that the initial equity of 50% for each player needs to climb to 100% for one of the players in order for the match to be finished. That much is clear. I also get that a swing as I described it would not bring the match to the centre of the curve since it cannot finish the match.

The dip reflects the fact that matches which have zero or little overall luck (per move) are rare. Close-to-zero luck means that winning the match was mostly due to skill.

Hence the "dipped bell shape" of the curve is a misleading expression. It consists rather of two bell shapes which are superimposed, with their high points at those good/bad luck values which are most common.

socksey

Quote from: Zorba on July 01, 2010, 09:31:35 PMWhat does "optimizing 'luck' potential" even mean? It sounds like a confused way of simply saying "maximizing equity".

Maybe it's an English thing!   :lol:

socksey



"The one thing I regret, is that I don't have more regrets." – Winston Churchill from his deathbed

blitzxz

Quote from: Zorba on July 01, 2010, 09:31:35 PM
What does "optimizing 'luck' potential" even mean?

I can tell what this means to me. You clear your mind and concentrate complitely to the flow of the game. Just before you or your opponent or computer rolls you clearly call the rolls you want in your head. And they start coming just the way you like them. I already showed how this works in the forum match 4.   ;) I'm sure this must be the reason why I have positive luck in gnu analyzes.

It's also important to call only good rolls for yourself and not bad ones for opponent because this could have negative effect on your aura.

stog

funny how if we 'tune in' and concentrate, we play better :)

pck

So here's a puzzle:

I let gnu play one-pointers against itself with both players set to the same level of skill. The outcome of these matches should be completely luck dependent.

But the stats say otherwise: The "Luck adjusted result" was never 0%, but varied between 3 and 15% in my tests.

The Luck adjusted result is calculated as 50% - the luck difference of the players, which should be their skill difference, which should be zero.

Any ideas what's wrong with this picture?

boomslang

Two reasons why it isn't zero that come to mind:

1) The neural nets are not 'perfect'. With perfect nets you would expect that the equity of the current position (after your opponent moved and prior to your roll) would be equal to the expected equity after playing your 21 possible rolls correctly.  In practice this is not the case.  If the nets would be perfect there would be no need to set a bot to 2 plies.
I think this is the main reason why the luck adjusted result (luck in general, actually) should be handled with care.  Maybe if you re-analyse the match at higher plies (for analysis AND luck -- I believe it is command "set analysis luckanalysis plies 2") the luck adjusted result will be closer to zero?


2) If you analysed the match at a different level as the playing level of the 2 players it might be that both players did not make the same error total during that match. The difference in errors doesnt cancel out then.


I am sure there are more causes though!

blitzxz

Quote from: pck on July 02, 2010, 12:20:13 PM
So here's a puzzle:

I let gnu play one-pointers against itself with both players set to the same level of skill. The outcome of these matches should be completely luck dependent.

But the stats say otherwise: The "Luck adjusted result" was never 0%, but varied between 3 and 15% in my tests.

The Luck adjusted result is calculated as 50% - the luck difference of the players, which should be their skill difference, which should be zero.

Any ideas what's wrong with this picture?

Luck adjusted results are notoriously inaccurate in single games or matches. It's actually worthless number to me. I once tried this same test playing, analyzing and analyzing luck with same ply and I think my record was over +50% luck adjusted result to other identical gnubg. I tried analyzing the same match (and luck also) with higher ply but it really didn't help, still the other bot was hugely more skillful. I can't really understand what is the cause of this. However in my long stats (fibs matches), luck adjusted results and unnormalized error rates seemed to converge.

pck

Quote from: blitzxz on July 02, 2010, 03:13:33 PM
Luck adjusted results are notoriously inaccurate in single games or matches. It's actually worthless number to me. I once tried this same test playing, analyzing and analyzing luck with same ply and I think my record was over +50% luck adjusted result to other identical gnubg. I tried analyzing the same match (and luck also) with higher ply but it really didn't help, still the other bot was hugely more skillful. I can't really understand what is the cause of this. However in my long stats (fibs matches), luck adjusted results and unnormalized error rates seemed to converge.

Analyzing with higher ply changed nothing for me either. Now it seems to me that this means that gnu's luck stats in general can be nothing more than a crude indication, since they are what the Luck adjusted result is calculated from. If this is true then vic has a lot of apologizing to do.

If the inaccuracies of gnu's luck calculation go both ways and are not systematically biased, they may of course even out in the long run as an effect of the process of averaging. This would indeed make the long stats more reliable than the single match-based ones.

pck

#50
Quote from: boomslang on July 02, 2010, 02:29:32 PM
1) The neural nets are not 'perfect'. With perfect nets you would expect that the equity of the current position (after your opponent moved and prior to your roll) would be equal to the expected equity after playing your 21 possible rolls correctly.  In practice this is not the case.  If the nets would be perfect there would be no need to set a bot to 2 plies.

I agree. What I had assumed was that non-perfect equity calculations should at least be internally consistent, that is, the numbers should add up. Obviously that is not the case. About the reason I still wonder. I have attached a gnu 1-pointer sgf file. What happens in the first two rolls is this:

Player "You" rolls a 45, gnu says that's an equity gain of +1.587% MWC. Player "gnubg" of course loses this amount of equity, as well as another 3.551% on its first roll, a 63. So gnubg's equity after rolling that 63 should be 50 - 1.587 - 3.551 = 44.862. But what it actually says is 45.77, a discrepancy of 0.908% after only one roll on each side. This is too large to be attributed to rounding errors. And if player gnubg's equity is calculated independently from previous equities, then why does gnu not adjust the -3.551 accordingly? Perhaps that would have other undesirable consequences I'm not seeing.

Quote from: boomslang on July 02, 2010, 02:29:32 PM
2) If you analysed the match at a different level as the playing level of the 2 players it might be that both players did not make the same error total during that match. The difference in errors doesnt cancel out then.

Good point. From a 3-ply point of view two 2-ply bot players will usually break symmetry with regard to their skill levels in a single match. Their errors should even out across a large number of matches though. Nevertheless this is very important for the consideration of what we call "skill". If we can attribute the term "skill" only to the whole of a bot's algorithm, that is, to its behaviour in all possible match situations, and if it could further happen that algorithm A1 beats A2 (in the long run), A2 beats A3, and A3 beats A1, then that would eliminate the possibility of attribution of skill for these three bots. Skill would only be a partial order on the set of all possible bot-algorithms. To prove that this is or isn't possible should be difficult.

sixty_something

Quote from: Zorba on July 01, 2010, 09:31:35 PM
pck's theoretical argument is correct and is actually pretty trivial from the definition of luck, for a statistician.

Quote from: pck on July 01, 2010, 10:55:33 PM
Yes, it's mathematically trivial because there is hardly any math involved! I was mainly concerned with giving an explanation of how we arrive at the confused notion that "skill begets luck".

Quote from: pck on July 03, 2010, 09:19:53 AM
What I had assumed was that non-perfect equity calculations should at least be internally consistent, that is, the numbers should add up. Obviously that is not the case. About the reason I still wonder.

is it safe to say we can drop the word "trivial" from your and Zorba's somewhat dismissive rebuttals of my suggestions?

i think you may be observing some of the same things i did when i first looked at "luck" calculatioins in detail ... like you, i "still wonder" about the reasons why what seems conceptually obvious simply does not add up when put to the test of striking a balance sheet for "luck" calculations

when observed values don't match theory and assumed concepts, perhaps it is time to revisit both with an open mind .. i am
A little inaccuracy sometimes saves tons of explanation. -- Unknown
e-mail me

blitzxz

Quote from: pck on July 03, 2010, 09:19:53 AM
Player "You" rolls a 45, gnu says that's an equity gain of +1.587% MWC. Player "gnubg" of course loses this amount of equity, as well as another 3.551% on its first roll, a 63. So gnubg's equity after rolling that 63 should be 50 - 1.587 - 3.551 = 44.862. But what it actually says is 45.77, a discrepancy of 0.908% after only one roll on each side. This is too large to be attributed to rounding errors. And if player gnubg's equity is calculated independently from previous equities, then why does gnu not adjust the -3.551 accordingly? Perhaps that would have other undesirable consequences I'm not seeing.

If I understand you correctly you are just adding up equity loses. It doesn't work that way. If you lose 1% first roll and 2% second it doesn't mean that now you have 47% chances to win. I haven't figured it out yet but I'm guessing some sort of multiplying is needed to add up equity loses and to come up with over all chances.

sixty_something

i'm with blitzxz on this one .. i've found using equity changes alone tends to yield expected additive values within a reasonalbe roundoff error .. what do the corresponding equity numbers look like?
A little inaccuracy sometimes saves tons of explanation. -- Unknown
e-mail me

pck

#54
Quote from: blitzxz on July 03, 2010, 12:20:08 PM
If I understand you correctly you are just adding up equity loses. It doesn't work that way. If you lose 1% first roll and 2% second it doesn't mean that now you have 47% chances to win. I haven't figured it out yet but I'm guessing some sort of multiplying is needed to add up equity loses and to come up with over all chances.

I'm not doing that. First I deduct the equity gain of the first player's initial roll from the second player's initial 50% (one player's gain is the other one's loss). From the result I deduct the equity (another loss due to a bad initial roll for player #2) from the second player's first roll and compare the total with what gnu says about the total equity of player #2 after its first roll.

If you load the attached sgf into gnu you will see it. I can't see how any multiplying would be involved here.

pck

Quote from: sixty_something on July 03, 2010, 11:32:53 AM
is it safe to say we can drop the word "trivial" from your and Zorba's somewhat dismissive rebuttals of my suggestions?

My "mathematically trivial" was not intended to be dismissive of your remarks. It was intended to mitigate Zorba's remark which sounded as if what I tried to argue in #30 was mainly concerned with technicalities. It wasn't. But I explained that right afterwards in #41.

Quote from: sixty_something on July 03, 2010, 11:32:53 AM
i think you may be observing some of the same things i did when i first looked at "luck" calculatioins in detail ... like you, i "still wonder" about the reasons why what seems conceptually obvious simply does not add up when put to the test of striking a balance sheet for "luck" calculations
when observed values don't match theory and assumed concepts, perhaps it is time to revisit both with an open mind .. i am

Couldn't agree more!

sixty_something

Quote from: pck on July 03, 2010, 06:01:19 PM
My "mathematically trivial" was not intended to be dismissive of your remarks.

no problemo, amigo

i suspect we can all get thin skinned when staking out a new and still controverial opinion - i know i can .. further, it never ceases to amaze me how easy it is to misinterpret even the most carefully crafted remarks in posts like thiese .. in shouts at FIBS it is even worse .. there a silly off hand comment or well intentioned insult amongst friends can too often lead to all out shout war and hurt feelings
so it goes in this zany little world of FIBSoids :unhappy:
A little inaccuracy sometimes saves tons of explanation. -- Unknown
e-mail me

Zorba

#57
Quote from: pck on July 01, 2010, 10:55:33 PM
Yes, it's mathematically trivial because there is hardly any math involved! I was mainly concerned with giving an explanation of how we arrive at the confused notion that "skill begets luck". What I wrote in #30 is not a theoretical argument (for it can neither be verified nor falsified), but an attempt at a conceptual elucidation by looking at how "luck" and "skill" are used in everyday language and how these terms are related to their mathematical counterparts.

I simply mean the theoretical (mathematical) argument that luck will tend towards zero in the long run, regardless of a player's skill. That can be verified quite easily using mathematics.

Quote
There is no a priori concept of "the correct definition of luck" for us to discover.

No, but this is not about philosophy. Once you've settled for a definition, like the bots have and which has been explained here and elsewhere, there are correct and false conclusions you can draw from it.

Most everything that's being discussed here is described very well in these articles by Douglas Zare:

http://www.bkgm.com/articles/Zare/AMeasureOfLuck.html
http://www.bkgm.com/articles/Zare/HedgingTowardSkill.html
The fascist's feelings of insecurity run so deep that he desperately needs a classification of some things as successful or superior and other things as failed or inferior. This also underlies the fascist's embracement of concepts like mental illness and IQ tests.  - R.J.V.

Luck is my main skill

Zorba

Quote from: pck on July 02, 2010, 01:39:11 AM
Hence the "dipped bell shape" of the curve is a misleading expression. It consists rather of two bell shapes which are superimposed, with their high points at those good/bad luck values which are most common.

I don't know if it that's true, it seems very hard to find out what kind of distribution of net luck per move really underlies the graph.
The fascist's feelings of insecurity run so deep that he desperately needs a classification of some things as successful or superior and other things as failed or inferior. This also underlies the fascist's embracement of concepts like mental illness and IQ tests.  - R.J.V.

Luck is my main skill

Zorba

Quote from: pck on July 02, 2010, 12:20:13 PM
So here's a puzzle:

I let gnu play one-pointers against itself with both players set to the same level of skill. The outcome of these matches should be completely luck dependent.

But the stats say otherwise: The "Luck adjusted result" was never 0%, but varied between 3 and 15% in my tests.

The Luck adjusted result is calculated as 50% - the luck difference of the players, which should be their skill difference, which should be zero.

Any ideas what's wrong with this picture?

Yes :)

Just because GnuBG 0-ply plays both sides, does not mean it plays both sides equally well in individual games. Only in the long run will they show equal skill levels. So the outcome of a particular match is not completely luck dependent: one side may have given up much more equity in errors than the other side.

Furthermore, GnuBG's luck evaluations, just like its error evaluations, are not perfect, so this is another factor contributing to inaccuracies, especially noticeable in the short run.

BTW, for practical purposes: GnuBG defaults to using 0-ply for its luck calculations. Luck calculations can be considered more difficult than normal evaluations, as 21 different dice rolls and their best plays have to be considered, so it's no surprise that 0-ply luck analysis can give rather inaccurate results generally. Use GnuBG's command line and the command boomslang mentioned to increase the ply level of GnuBG's luck analysis.

Another interesting thing to consider here is that a n-ply luck analysis is closer to a (n+1)-ply error analysis than to a n-ply error analysis, due to the 21 different rolls that have to be analyzed. This is also true for the time it takes to do such an analysis: using 2-ply for luck analysis is about as slow as doing a 3-ply error analysis.
The fascist's feelings of insecurity run so deep that he desperately needs a classification of some things as successful or superior and other things as failed or inferior. This also underlies the fascist's embracement of concepts like mental illness and IQ tests.  - R.J.V.

Luck is my main skill