AttentiveNAS<T>

Models & Types · AiDotNet.AutoML.NAS

AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling. Uses an attention-based meta-network to guide the sampling of sub-networks, focusing search on promising regions of the architecture space.

Reference: “AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling” (CVPR 2021)

Inheritance: AutoMLModelBase<T, Tensor<T>, Tensor<T>> → NasAutoMLModelBase<T> → AttentiveNAS<T>

For Beginners

AttentiveNAS improves architecture search by paying attention to which sub-networks perform well. Instead of sampling random architectures, it learns to focus on promising designs, like a smart student who concentrates on the most important study topics rather than studying everything equally.

Example

var searchSpace = new SearchSpaceBase<float>();
var nas = new AttentiveNAS<float>(searchSpace);
Architecture<float> architecture = nas.BestArchitecture;

Constructors

Constructor Summary
AttentiveNAS(SearchSpaceBase<T> searchSpace, List<int> elasticDepths, List<double> elasticWidthMultipliers, List<int> elasticKernelSizes, int attentionHiddenSize)

Properties

Property Summary
AttentiveSamplingStrategy SamplingStrategy { get; set; } Gets or sets which candidate attentive sampling trains: the predicted Pareto-best (BestUp, the default) or Pareto-worst (WorstUp) of the pool drawn under a FLOPs target (Wang et al.
int CandidatePoolSize { get; set; } Gets or sets k, the number of candidates drawn under each FLOPs target before the best or worst is picked (default 3).

Methods

Method Summary
AttentiveNASConfig<T> AttentiveSample(Vector<T> contextVector) Samples an architecture by attentive sampling (Wang et al.
AttentiveNASConfig<T> Search(HardwareConstraints<T> constraints, int inputChannels, int spatialSize, int numIterations) Searches for optimal architecture using attentive sampling
Dictionary<string, T> GetPerformanceMemory() Gets the performance memory
Tensor<T> GetAttentionWeights() Gets the attention weights
Vector<T> CreateContextVector() Creates a context vector from recent architecture performance history
void UpdateAttention(AttentiveNASConfig<T> config, T performance, T learningRate) Updates the attention module based on architecture performance: one gradient step of the predictor on the squared error (prediction - performance)^2 / 2, i.e.