TCAVExplainer<T>

Models & Types · AiDotNet.Interpretability.Explainers

Testing with Concept Activation Vectors (TCAV) explainer.

Implements: IGPUAcceleratedExplainer<T>

For Beginners

TCAV is a technique that explains model predictions using human-understandable concepts instead of raw features. While most explanation methods tell you which pixels or features matter, TCAV tells you which CONCEPTS matter.

Example: Instead of highlighting pixels that matter for a “doctor” prediction, TCAV can tell you whether the concept of “stethoscope” or “white coat” influenced the prediction.

How TCAV works:

  1. Collect concept examples: Gather images/examples that represent your concept

(e.g., images with stripes for a “striped” concept)

  1. Collect random examples: Gather examples that don’t specifically represent the concept
  2. Train a CAV: Train a linear classifier to distinguish concept vs random at a

specific layer. The classifier’s weight vector becomes the Concept Activation Vector (CAV).

  1. Compute directional derivatives: See how sensitive the model output is to moving

in the direction of the CAV

  1. Compute TCAV score: The fraction of test inputs where moving toward the concept

increases the model’s prediction for a class

Interpreting TCAV scores:

  • TCAV score = 0.8 means 80% of images are more likely to be classified as the target

class when they have more of the concept

  • TCAV score = 0.5 means the concept doesn’t influence the prediction (random)
  • TCAV score = 0.2 means the concept DECREASES the likelihood of the target class

Statistical significance: TCAV runs multiple times with different random samples to ensure the score is statistically significant and not due to random noise.

When to use TCAV:

  • When you want high-level, human-understandable explanations
  • When you have examples of concepts you want to test
  • For testing fairness/bias (e.g., does “gender” concept affect hiring predictions?)
  • For debugging models (e.g., is my model relying on spurious correlations?)

Constructors

Constructor Summary
TCAVExplainer(Func<Vector<T>, Vector<T>> predictFunction, Func<Vector<T>, Vector<T>> layerActivationFunction, Func<Vector<T>, int, Vector<T>> gradientToLayerFunction, int layerSize, double regularization, int numCavs, double significanceLevel, int? randomState) Initializes a new TCAV explainer.

Properties

Property Summary
bool IsGPUAccelerated { get; }
bool SupportsGlobalExplanations { get; } Gets whether this explainer supports global explanations.
bool SupportsLocalExplanations { get; } Gets whether this explainer supports local explanations.
string MethodName { get; } Gets the method name.

Methods

Method Summary
ConceptActivationVector<T> TrainCAV(Matrix<T> conceptExamples, Matrix<T> randomExamples) Trains a Concept Activation Vector from concept and random examples.
LocalTCAVExplanation<T> ExplainLocal(Vector<T> input, ConceptActivationVector<T> cav, int targetClass, string conceptName) Computes concept sensitivity for a single input.
T ComputeDirectionalDerivative(Vector<T> input, ConceptActivationVector<T> cav, int targetClass) Computes the directional derivative of the model output with respect to the CAV.
TCAVExplainer<T> FromNetwork(INeuralNetwork<T> network, Func<Vector<T>, Vector<T>> layerActivationFunction, int layerSize, int numCavs, double significanceLevel, int? randomState) Creates a TCAV explainer from a neural network using input gradient helper.
TCAVResult<T> RunTCAV(Matrix<T> conceptExamples, Matrix<T> randomExamplesPool, Matrix<T> testInputs, int targetClass, string conceptName) Runs a full TCAV analysis with statistical significance testing.
TCAVResults<T> RunTCAVMultiple(Dictionary<string, Matrix<T>> concepts, Matrix<T> randomExamplesPool, Matrix<T> testInputs, int targetClass) Runs TCAV analysis for multiple concepts.
double ComputeTCAVScore(Matrix<T> testInputs, ConceptActivationVector<T> cav, int targetClass) Computes the TCAV score for a concept on a set of inputs.
void SetGPUHelper(GPUExplainerHelper<T> helper)