PIQABenchmark<T>

Models & Types · AiDotNet.Reasoning.Benchmarks

PIQA (Physical Interaction Question Answering) benchmark for physical commonsense reasoning.

Implements: IBenchmark<T>

For Beginners

PIQA tests whether AI understands how the physical world works through everyday situations and questions about physical interactions.

What is PIQA? PIQA asks questions about physical commonsense - understanding how objects interact, what happens when you do certain actions, and basic physics of everyday life.

Format:

  • Goal: A task you want to accomplish
  • Solutions: Two possible ways to do it (one correct, one wrong)
  • Question: Which solution actually works?

Example 1:

Goal: To separate egg whites from the yolk
Solution 1: Crack the egg into a bowl, then use a water bottle to suck up the yolk
Solution 2: Crack the egg and use your hands to throw the white away
Correct: Solution 1

Example 2:

Goal: To remove a stripped screw
Solution 1: Place a rubber band over the screw head for better grip
Solution 2: Pour water on the screw to make it easier to turn
Correct: Solution 1

Example 3:

Goal: Keep your garbage can from smelling bad
Solution 1: Spray perfume in the garbage can every day
Solution 2: Put baking soda at the bottom of the can
Correct: Solution 2

Why it’s important:

  • Tests real-world physical understanding
  • Can’t be solved by language patterns alone
  • Requires knowledge of:
  • Basic physics (gravity, friction, pressure)
  • Material properties (hard, soft, sticky, etc.)
  • Cause and effect in physical world
  • Practical life skills

Performance levels:

  • Random guessing: 50%
  • Humans: 94.9%
  • BERT: 70.9%
  • RoBERTa: 79.4%
  • GPT-3: 81.0%
  • GPT-4: 86.8%
  • Claude 3 Opus: 85.2%
  • Claude 3.5 Sonnet: 88.0%
  • ChatGPT o1: 91.5%

Categories:

  • Kitchen tasks
  • Home repair
  • Cleaning
  • Arts and crafts
  • General household

Why it’s hard for AI:

  • LLMs lack embodied experience
  • Can’t actually touch or manipulate objects
  • Must infer from text descriptions
  • Requires implicit physical knowledge

Research:

  • “PIQA: Reasoning about Physical Commonsense in Natural Language” (Bisk et al., 2020)
  • https://arxiv.org/abs/1911.11641
  • Dataset: 16,000 questions from WikiHow and other sources
  • Part of physical reasoning evaluation suite

Constructors

Constructor Summary
PIQABenchmark()

Properties

Property Summary
int TotalProblems { get; }
string BenchmarkName { get; }
string Description { get; }

Methods

Method Summary
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken)
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count)