HadaNets: Flexible Quantization Strategies for Neural Networks
- 0 Collaborators
Deep learning has proven to be effective for a variety of tasks. As the market for mobile and offline artificial intelligence develops, the need to reduce the memory and energy footprint of neural networks becomes vital. In this paper, we introduce HadaNets. HadaNets utilize a flexible tensor quantization scheme which trains a neural network from scratch by ‘pairing’ a full precision tensor to a binary tensor in the form of a Hadamard product. The train-time parameter count is preserved in HadaNets. On the ILSVRC12 ImageNet classification dataset we reduce memory consumption by 7.43 times (with respect to full precision models) while yielding improved accuracy over other binary neural networks. Our 'Hadamard Binary Matrix Multiply' kernel delivers a 10-fold increase in performance over full precision matrix multiplication with a similarly optimized kernel. ...learn more
Project status: Under Development
Groups
Student Developers for AI
Overview / Usage
• We introduce a quantized neural network training strategy with flexible memory and energy requirements. HadaNets yield a higher accuracy than XNOR-Nets without increasing filter map counts, as opposed to WRPN Nets.
• Binary neural networks require us to maintain fullprecision parameters for gradient updates, making HadaNets the most effective strategy to train quantized neural networks from scratch. Our training methodology performs at par with post-training quantization
methodologies like ABC-Nets (±0.5% top-1 accuracy).
• We introduce Hadamard-Binary-Weight-Networks (HBWNs) (indicated by βa = 1 in our tests). HBWNs outperform Binary-Weight-Networks [23] on the ResNet-18 topology by 1.5% in top-1 accuracy. HBWNs ourperform post-training weight quantization methodologies such as Network Sketching with the AlexNet network topology.
• We develop Hadamard binary matrix multiply CPU kernel which delivers a 10 fold increase in performance over its full precision counter-part.
Technologies Used
Intel AI DevCloud: Intel Xeon Gold 6128
OpenMP
PyTorch
CUDA
Repository
https://github.com/akhauriyash/HadaNet/