While I was analyzing the data for a board game, a very interesting and important issue came up. In all kinds of gaming data, it's always about two things.
1. What moves did a player make during the game?
2. Did that player win?
With modern progresses of machine learning, there are many different ways to look at the data. However, doesn't matter how we look at the data, anything we can directly get from this type of data is conditional probability. Namely,
The probability to win given a certain move.
The inconvenient truth is that, this conditional probability has no definite connection to the intrinsic value of such move. Colloquially, the reason is very simple. Whenever we see a move with a high conditional winning probability, we never know which the following 2 cases is true:
(1) This move really helps you win.
(2) Good players prefer to make this move, and good players win more often.
Unless all players who showed up in the data are exactly equally skilled, the conditional winning probability always come from the combination of these two effects. In order to get the intrinsic value of such move, we need to isolate effect (1). Thus, we will be referring to effect (2) as the ``skill bias'' and attempt to remove it. Also, the conditional probability is something easily computable from the data. We would hope that the removal of skill bias is not much more complicated.
To my surprise, I asked this question on stat.stackexchange but no one pointed me to any really relevant existing literature. I also asked a few Math/Econ researchers and none of them is particularly aware of this topic. So, I decide to solve this problem myself with a simple Mathematical model.
Long story short, to my pleasant surprise, I got a somewhat simple answer.
Intrinsic Value of a Move = (P1 - P2)/(1-8d^2)
P1 is just the conditional probability to win given this move.
P2 is more subtle. It is the expected chance for someone from Group 1 to defeat someone from Group 2. Group 1 and 2 can be thought of as random subsets with the following selection rule:
There are 100 games in the data, player A appears in 17 of them, made this move in 8 games, and did not make this move in the other 9 games. Player A has a 8% chance to be selected from Group 1, and 9% chance to be selected from Group 2.
The readers can probably appreciate that P2 is trying to calculate contribution from effect (2) and allows us to remove it. Acute readers will also notice that such removal will result in an under-estimation. That is because part of the advantage of being a good player is that they make this move more often. Removing that altogether is removing part of the intrinsic value of this move. That is why after the subtraction, we divide the answer by a number that is slightly smaller than 1. d is a small number that I won't explain here. Please read my paper if you are interested. The paper also include instructions on how to calculate P2 and d in general.
The End of Group Thinking?
From their definitions, it is not guaranteed that P1 > P2. In other words, it is possible for a bad move (Intrinsic Value < 0) to have a high conditional winning probability (>50% in 2er games), simply because good players prefer to make this move (because they mistook this move as being good). Without a proper removal of skill bias, it is really difficult to recognize this situation. Good players will continue to make those bad moves that they consider good, and the conditional winning probabilities of those move will be high. It will appear convincing as if those moves were good.
This self-fulfilling prophecy of good moves is one of the main reasons that Group Thinking occurs. It is fortunate that by removing a skill bias, we have a way to directly combat group thinking.
In fact, Group Thinking is also nurtured by our habit of "learning form the experts". We listen to advices from good players, watch and mimic their moves. Therefore, we inherit their biases. One amazing property of the Intrinsic Value is that it does not care about the average skill of players in the data! You can do it with a group of experts, or a group of mediocre players. You will get the same answer. Such an objective approach can help us to get rid of existing biases.
