Large language models (LLMs) trained on massive corpora of human knowledge show strong abilities in reasoning, question answering, and knowledge synthesis. Linked to large-scale scholarly records, they can approximate scientific trajectories and plausible alternatives. We present a framework that simulates scholarly conversations among multi-agent LLMs instantiated as digital twins of real researchers, each grounded in publication histories. Modeling 8,269 authors of 2024 papers across nine flagship computer-science conferences, agents propose and refine hypothetical 2024 research ideas beyond the models’ knowledge cutoff. Generated ideas are compared with actual papers using semantic-similarity and convex-hull analyses at the author and conference levels. At the author level, cross-classification shows strong correspondence between generated and focal classifications (94.4% inside–inside; 85.3% outside–outside), indicating that simulations approximate researchers’ research directions. At the conference level, simulations produced at least one idea outside both 2023 and 2024 convex hulls for 99.1% of focal papers, but these ideas were more recombinatorial and less feasible than human-authored ones. Overall, the results suggest that LLM-based simulations mirror established patterns of scientific production but tend to extend into less immediately feasible directions, recombining existing ideas.

This paper calls on the research community not only to investigate how human biases are inherited by large language models (LLMs) but also to explore how these biases in LLMs can be leveraged to make society's "unwritten code" - such as implicit stereotypes and heuristics - visible and accessible for critique. We introduce a conceptual framework through a case study in science: uncovering hidden rules in peer review - the factors that reviewers care about but rarely state explicitly due to normative scientific expectations. The idea of the framework is to push LLMs to speak out their heuristics through generating self-consistent hypotheses - why one paper appeared stronger in reviewer scoring - among paired papers submitted to 45 academic conferences, while iteratively searching deeper hypotheses from remaining pairs where existing hypotheses cannot explain. We observed that LLMs' normative priors about the internal characteristics of good science extracted from their self-talk, e.g., theoretical rigor, were systematically updated toward posteriors that emphasize storytelling about external connections, such as how the work is positioned and connected within and across literatures. Human reviewers tend to explicitly reward aspects that moderately align with LLMs' normative priors (correlation = 0.49) but avoid articulating contextualization and storytelling posteriors in their review comments (correlation = -0.14), despite giving implicit reward to them with positive scores. These patterns are robust across different models and out-of-sample judgments. We discuss the broad applicability of our proposed framework, leveraging LLMs as diagnostic tools to amplify and surface the tacit codes underlying human society, enabling public discussion of revealed values and more precisely targeted responsible AI.