Let me try to review my notes from the past thread so I can move past them.
Overview
The cognitive lie detection approach outlined in the paper consists of three techniques:
- imposing cognitive load,
- encouraging interviewees to say more, and
- asking unexpected questions.
The initial paper and its 2017 critique find that this approach works around 60% of the time when humans are behind final decision-making about honesty. It's much higher when a formal process (i.e. an algorithm) is applied, success rates are a lot better. In a long text-based mafia game I think we might have the time and energy to achieve something closer to the ceiling set by formal approaches. If you do choose that framing (which I will going forward), then meta-analysis across studies finds a success rate approaching 80%.
Relevance for Mafia As We Play It
But then again, there's no way to know if it works as well in text-based mafia, where people get to prepare their posts, don't give off non-verbal cues, and aren't under the same sort of pressure as a lot of research conditions surrounding this technique foster.
And on the other hand, the method is pretty obvious from our perspective. Overall site meta already encourages people to use votes and questions to "pressure" people into producing content and giving off tells, etc. At the same time, I think a lot of players take a more passive approach to mafia than this research suggests is optimal. They read threads, produces some reads, maybe place down votes and then sort of check-out or press their case.
A better approach to sorting players along with generally putting pressure on them is probably to think up (ideally odd/unanticipated) questions that force them to elaborate
further
on stuff they've said/done in the thread. So like, rather than just making an accusation and watching someone squirm about it, actually interrogate them (along with making them squirm, yeah). But again, that's probably quite obvious to anyone who's played enough of this game.
Nonetheless, the medium by which we play this game makes this approach less likely to succeed though. In particular, people have room to workshop their answers, making cognitive load pretty hard to impose unless players are forced to respond immediately somehow. Alternatively, maybe time taken to produce a response becomes something worth tracking here. However, it's a very noisy variable and other research finds that hesitation isn't a good marker of lying anyway. Perhaps a combination of reaction time tracking and pressuring interviewees to respond promptly could maximize the approach's success here. Tough concept to test though!
But What Comes Next After Pressure?
More broadly, something missing from the "cognitive approach to lie detection" meta-analysis referenced above though is a direct account of what people should be looking for when they try the technique? What is it that people do when you apply this technique that differentiates truth-tellers and liars? There are some explanations of some of this in the cited paper, but they're pretty terse. Even the table in the spoilered tweet leaves mysterious how any classification scheme could have 60-80% accuracy in this context.
Wht I do see seems to depend on noticing that someone is
struggling
- i.e. to promptly generate accurate details about an event they should have familiarity with if it actually happened. I can see how that's a useful cue in a face-to-face interview where someone's gotta answer your questions immediately and you have the authority to impose cognitive load in odd ways like having them tell a story backwards, but tons of factors make this super hard in text-based games.
So atm I feel like the main themes of this paper can tell us about one piece of the equation (how to treat suspects so they give off tells) but not the other parts: what tells actually
look
like, and how to generalize this stuff to a meaningfully different format.
The studies within the meta-analysis applying a formal/algorithmic approach for identifying liars probably deserve the most attention going forward because they'll identify concrete features in participants' responses that contribute to these high success rates - i.e., tells. They'll also offer insight into how to properly aggregate these features to form a decision, an issue that's received less attention in this community, but is potentially just as fraught.
Reviewing those might provide actionable advice to people about how to sort players in their games. And they might give us a firmer basis for statistical projects people in this community might be interested in trying. So I'll look into those next.
![:]](./images/smilies/icon_smiling.png "Cheerful Smile"){kind=icon}