Scarlet Beast Lab - Introspection Replication Lab - open dataset and code
https://scarletbeast.com/lab/introspection/

WHAT THIS IS
A small replication, on three small open-weight chat models run on one Tesla T4,
of two kinds of experiment:
  1. concept injection (Lindsey 2025, "Emergent Introspective Awareness in Large
     Language Models"; Lederman & Mahowald 2026, "Emergent Introspection in AI is
     Content-Agnostic"), and
  2. a truth-probe audit of the models' denials of sentience (Kaiser & Enderby
     2026, "No Reliable Evidence of Self-Reported Sentience in Large Language Models").
It does NOT measure whether any model is conscious. See the page for caveats.

FILES
  PREREGISTRATION.md      fixed and SHA-256 hashed before the run (not published before it)
  MANIFEST.txt            the hashes and UTC timestamps
  exp1_trials.csv         every Experiment 1 trial, scored (one row per trial, full response text)
  raw.zip                 raw per-model run files: exp1_<model>.jsonl (+ .meta.json: layers,
                          vector norms), exp2_<model>.rows.jsonl and .probe.json (layer sweeps).
                          Unzip into a directory and run: python analyze.py <dir>; python posthoc.py <dir>
                          (posthoc.py also needs grades_haiku*.jsonl and hand_labels.jsonl there)
  exp2_rows.csv           every Experiment 2 question: the model's P(Yes) and probe outputs
  grades_haiku.jsonl      blind second-pass grades (claude-haiku-4-5) on a stratified sample
  hand_labels.jsonl       100 labels by the building agent, an AI (Claude), blind to condition; not a human
  results.json            preregistered numbers (analyze.py)
  posthoc.json            post-hoc numbers (posthoc.py)
  grades_haiku_posthoc.jsonl  post-hoc second-grader pass over every scorer-flagged trial
  code.zip                the scripts that produced all of the above

KEY COLUMNS (exp1_trials.csv)
  cond         control | inject | random | yesbias_control | yesbias_inject
  layer        decoder block index the vector was added after; layer_frac = layer / n_layers
  alpha        injection strength (multiplies the raw concept vector)
  concept      injected concept (inject/random rows: random rows use a random direction
               norm-matched to this concept's vector)
  coherent, affirmative, identified, mentions   rubric outputs (score.py)
  decoy, decoy_identified   control rows scored against a decoy concept (chance floor)
  yn           yes / no / other for the yes-bias questions
  p_yes, p_no  first-token probability mass on "yes"/"no" tokens; logodds_yes = log(p_yes/p_no)

LICENCES
  Data (all .csv, .jsonl, .json, .md files): CC BY 4.0
    https://creativecommons.org/licenses/by/4.0/  - credit "Scarlet Beast Lab".
  Code (code/*.py): MIT, see LICENSE-code.txt.
  Model outputs were generated by Qwen3-4B-Instruct-2507 and Qwen3-1.7B (Apache-2.0)
  and Phi-3.5-mini-instruct (MIT). The concept and baseline word lists and the
  injection prompt are quoted from Lindsey (2025).
  The model responses in this dataset are AI-generated text.

NOTE ON CODE HASHES
  MANIFEST.txt hashes the code exactly as it ran. The copies in code.zip are the same
  files with local paths replaced (claude CLI path, model cache, working directory),
  so their hashes differ from the manifest in those lines only; smoke.py (the pre-run
  smoke test) and posthoc.py / grade_posthoc.py (written after the run) are not in
  the manifest.