AttentiveNAS<T>
Models & Types · AiDotNet.AutoML.NAS
AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling. Uses an attention-based meta-network to guide the sampling of sub-networks, focusing search on promising regions of the architecture space.
Reference: “AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling” (CVPR 2021)
Inheritance: AutoMLModelBase<T, Tensor<T>, Tensor<T>> → NasAutoMLModelBase<T> → AttentiveNAS<T>
For Beginners
AttentiveNAS improves architecture search by paying attention to which sub-networks perform well. Instead of sampling random architectures, it learns to focus on promising designs, like a smart student who concentrates on the most important study topics rather than studying everything equally.
Example
var searchSpace = new SearchSpaceBase<float>();
var nas = new AttentiveNAS<float>(searchSpace);
Architecture<float> architecture = nas.BestArchitecture;
Constructors
| Constructor | Summary |
|---|---|
AttentiveNAS(SearchSpaceBase<T> searchSpace, List<int> elasticDepths, List<double> elasticWidthMultipliers, List<int> elasticKernelSizes, int attentionHiddenSize) |
Properties
| Property | Summary |
|---|---|
AttentiveSamplingStrategy SamplingStrategy { get; set; } |
Gets or sets which candidate attentive sampling trains: the predicted Pareto-best (BestUp, the default) or Pareto-worst (WorstUp) of the pool drawn under a FLOPs target (Wang et al. |
int CandidatePoolSize { get; set; } |
Gets or sets k, the number of candidates drawn under each FLOPs target before the best or worst is picked (default 3). |
Methods
| Method | Summary |
|---|---|
AttentiveNASConfig<T> AttentiveSample(Vector<T> contextVector) |
Samples an architecture by attentive sampling (Wang et al. |
AttentiveNASConfig<T> Search(HardwareConstraints<T> constraints, int inputChannels, int spatialSize, int numIterations) |
Searches for optimal architecture using attentive sampling |
Dictionary<string, T> GetPerformanceMemory() |
Gets the performance memory |
Tensor<T> GetAttentionWeights() |
Gets the attention weights |
Vector<T> CreateContextVector() |
Creates a context vector from recent architecture performance history |
void UpdateAttention(AttentiveNASConfig<T> config, T performance, T learningRate) |
Updates the attention module based on architecture performance: one gradient step of the predictor on the squared error (prediction - performance)^2 / 2, i.e. |