This research programme began with the “Humans in Humans Out” project, which evaluated large language models (LLMs) on reasoning patterns, both accurate and fallacious, that resemble human thinking. The initial work assessed whether LLMs replicated predictable human-like reasoning errors on a relatively small corpus of reasoning problems with empirical support in the literature. The goal was to uncover whether LLMs might converge toward common-sense reasoning patterns: mimicking both human successes and human failures at scale, potentially informing novel training strategies while shedding light on what aspects of human reasoning are being approximated by LLMs trained on large human-generated datasets.
Recently, the programme has evolved into a more systematic treatment grounded in the Erotetic Theory of Reasoning (ETR), which formalises how people ask and answer questions and predicts both correct and fallacious patterns of inference. In our new study “Stronger Language Models Produce More Human-Like Errors” we used ETR-based problems to demonstrate that as language models improve, their mistakes increasingly resemble human fallacies. To enable broader exploration of this line of work, HAILab has released the open-source ETR Case Generator, a tool built on our PyETR library that creates large banks of test cases for analysing reasoning in humans and AI systems alike. The package makes it possible for philosophers and cognitive scientists to explore and extend this research directly from code.
Download the ArXiv paper on preliminary work here.
