Verbatim Index

What is being scored

Each transcript is graded on three dimensions, each on a 1–100 scale, supported by 15 sub-criteria that a judge may also mark not observed when the format gave no opportunity to demonstrate them. The Overall score is 0.45×Insight + 0.35×Technical + 0.20×Clarity.

• Insight (45%) — is the thinking any good? Causal and counterfactual reasoning, originality, strategic tradeoffs, understanding of incentives, calibration of confidence, engagement with opposing arguments, and whether the speaker recomputes when a premise changes. It carries the most weight because it is what the study is asking about.
• Technical and industry depth (35%) — does the speaker actually know how their world works? This is not an engineering test, and only one of its four parts is technical in that sense. A non-engineer who genuinely knows how their business runs scores well here.
  • How the thing works — architecture, constraints, failure modes; technical terms used with their real meaning rather than as decoration.
  • Numbers — magnitudes, rates, denominators and unit economics, used correctly, and knowing which ones actually bear on the decision.
  • How the industry actually works — the part people underrate. Where the bottleneck really is, how the supply chain behaves, what the sales motion looks like, what the org will and will not do, which regulator has to say yes. Naming the two suppliers who can make a part, or why a deal closes in nine months rather than three, counts here.
  • Moving between layers — going from business objective to user behaviour to product requirement to technical constraint to organisational decision without losing the thread.
  The hinge is whether the detail changes the conclusion rather than decorating it. Reciting a specification scores low; explaining why that specification forced a particular choice scores high. Jargon density, name-dropping technologies and confident claims that are wrong on their face are all explicitly not rewarded. Depth carries the second-largest weight because it is what makes insight load-bearing rather than rhetorical.
• Clarity (20%) — can a listener follow it? Directness, structure, concrete language, economy. Deliberately the smallest weight: the rubric explicitly refuses to reward fluency, charisma, or a confident delivery, because a polished non-answer is the failure mode being guarded against.

The pipeline

• A roster of 53 was built by 12 parallel sector scouts pooling 240 candidates, merged, then attacked by four adversarial critics checking fame, availability, factual accuracy, and coverage bias. Contested cases were settled by measuring actual long-form supply, not by argument.
• Transcripts are verbatim automatic captions of publicly posted recordings, pulled programmatically with timestamps. No model paraphrased or summarised them at any point.
• Every transcript passed deterministic quality gates before grading: out-of-vocabulary rate, speech-recognition repetition loops, type-token ratio, words per minute, and minimum length.
• Speaker and company names were replaced with [SUBJECT] and [COMPANY], including speech-recognition manglings of the surname found by phonetic matching. This step is covered by tests, after an early version replaced the contraction “that’s” with [SUBJECT] 34 times in one transcript.
• Three judges graded each transcript independently, each at its highest reasoning setting, and no judge ever sees another's answer: OpenAI GPT-6 Astra (gpt-6-astra, 676 transcripts); Claude Fable 5.1 (claude-fable-5-1, 685 transcripts); Google Gemini 3.8 Flash (gemini-3.8-flash-high, 650 transcripts). The model that served each call was read from that call's own telemetry rather than assumed.

How reliable is a score?

One unchanged transcript was graded five times by Fable and five times by Astra under identical conditions. The spread that produced is pure method noise.

• Repeat grading moves the Overall score by about 1.57 points of standard deviation.
• So two leaders differing by less than 4.3 points are not distinguishable. The bracket in the rank column marks those groups.
• Coverage and venue-difficulty judgements were perfectly stable across repeats (standard deviation 0.00), so the judges read the same conversation the same way every time.

How much of this is measurement, and how much is noise

Every figure below was measured on this corpus, not assumed.

• Each judge repeats itself within about 2 points. Grading the same transcript twice gave a mean absolute difference of 1.69 (Fable) and 2.06 (Astra), implying a single-grading standard deviation near 1.64. Measured two independent ways, five repeats on one fixture and nine pairs across four different transcripts, which agree.
• Fable and Astra differ from each other by 8.65 points, which is four to five times either judge's own noise. So where they disagree, that is a real difference of opinion about the transcript and not instability in either model. Agreement on the ranking is ICC(3,1) = 0.657, moderate on the Koo and Li bands, and the 95% limits of agreement span 30 points. Read the ranking as the shape of the field, never as a verdict on one appearance.
• Length does not buy score. Twelve padded variants of one transcript, three padding styles at +25% and +60% words, moved the Overall score by +0.05 (Fable) and −1.08 (Astra), every one inside the noise floor. Astra's clarity score fell 5.7 on padded text while insight and technical depth held flat, which is the rubric working as designed.
• Fame does not buy score. Correlation between a video's view count and its Overall score is +0.093, against the 0.312 needed for significance at this sample size.
• The three dimensions measure three things. In a multitrait-multimethod matrix, every same-dimension cross-judge correlation (0.661 to 0.686) beats every different-dimension pair (0.260 to 0.599).
• Being told who is speaking barely moves the score. Across 28 paired blinded and unblinded gradings the halo is +0.48 points, inside two standard errors of zero.

What this cannot tell you

• Blinding removed the name, not the identity. This matters less than it sounds: the measured halo above is +0.48 points, inside noise. Judges recognised the speaker anyway in 100% of blinded transcripts, from products, projects, and context. Defeating that would mean stripping the technical content the study exists to measure. 233 of the 686 transcripts were therefore graded a second time unblinded, and the Halo column reports the gap on those: how many points a leader gains once the judges are told who they are. The published Overall score is always the blinded one.
• Captions have no speaker labels. Judges separated the subject's speech from the interviewer's by context and reported their confidence and the subject's estimated share of the talking. Both are shown in each transcript card.
• Venue format is corrected for. Venue difficulty is not. Leaders are not spread evenly across formats, so the published score subtracts what the format is worth with the speaker held fixed, putting a keynote and a long-form podcast on the same footing. What is never corrected for is how hard the conversation itself was. A leader who only sits for friendly interviews will score lower on insight, because a soft conversation cannot demonstrate reasoning under pressure. Adjusting for that would mean inventing a score for a conversation that never happened, so each transcript reports its judged difficulty instead.
• Sampling is not exhaustive. A handful of appearances per leader is a sample of a public speaking record, not the whole of it. Leaders marked low confidence have too few transcripts for their rank to be trusted.
• Not every transcript carries every judge. The panel grew to three judges partway through the corpus, so 639 of 686 transcripts hold all 3 opinions. Each transcript is scored on the grades that exist for it. No missing grade is ever estimated or filled in.
• Reliability was measured on Fable and Astra. The repeat-noise, tie-band and agreement figures above come from those two arms, before the third joined. They are not yet recomputed for the full panel.
• Judges are language models. Three frontier models with different training agreeing on a ranking is meaningful evidence, and it is not the same thing as being correct.

Who was excluded, and why

Excluded names are recorded rather than silently omitted, because silence looks like an oversight. 11 people famous enough for consideration were left out, most for lack of retrievable long-form public speech: John Ternus, Liang Wenfeng, Lei Jun, Hock Tan, Masayoshi Son, Shou Zi Chew, Jony Ive, Michael Truell, Larry Page, Ali Ghodsi, Matt Garman.