Abstract
This article presents a scalable deep learning model called Agile Temporal Convolutional Network (ATCN) for highly accurate fast classification and time series prediction in resource-constrained embedded systems. ATCN is a family of compact networks with formalized hyperparameters that enable application-specific adjustments to be made to the model architecture. It is primarily designed for embedded edge devices with very limited performance and memory, such as wearable biomedical devices and real-time reliability monitoring systems. ATCN makes fundamental improvements over the mainstream temporal convolutional neural networks, including residual connections to increase the network depth and accuracy and the incorporation of separable depth-wise convolution to reduce the computational complexity of the model. As part of the present work, two ATCN families, namely T0 and T1, are also presented and evaluated on different ranges of embedded processors: Cortex-M7 and Cortex-A57 processors. An evaluation of the ATCN models against the best-in-class InceptionTime and MiniRocket shows that ATCN almost maintains accuracy while improving the execution time on a broad range of embedded and cyber-physical applications with demand for real-time processing on the embedded edge. At the same time, in contrast to existing solutions, ATCN is the first time series classifier based on deep learning that can be run bare-metal on embedded microcontrollers (Cortex-M7) with limited computational performance and memory capacity while delivering state-of-the-art accuracy.
- [1] . 2019. Real-time deep learning at the edge for scalable reliability modeling of Si-MOSFET power electronics converters. IEEE Internet of Things Journal 6, 5 (2019), 7375–7385.
DOI: Google ScholarCross Ref
- [2] . 2021. DeepDive: An integrative algorithm/architecture co-design for deep separable convolutional neural networks. In Proceedings of the 2021 on Great Lakes Symposium on VLSI (GLSVLSI’21). Association for Computing Machinery, New York, NY, 247–252.
DOI: Google ScholarDigital Library
- [3] . 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. CoRR abs/1803.01271 (2018). arXiv:1803.01271 http://arxiv.org/abs/1803.01271.Google Scholar
- [4] . 2018. Scalable reliability monitoring of GaN power converter through recurrent neural networks. In 2018 IEEE Energy Conversion Congress and Exposition (ECCE’18). 7271–7277.
DOI: Google ScholarCross Ref
- [5] . 2020. Optimizing temporal convolutional network inference on FPGA-based accelerators. IEEE Journal on Emerging and Selected Topics in Circuits and Systems (2020), 1–1.Google Scholar
- [6] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. 2014. Empirical evaluationof gated recurrent neural networks on sequence modeling. In NIPS DeepLearn Workshop.Google Scholar
- [7] . 2019. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica 6, 6 (2019), 1293–1305.
DOI: Google ScholarCross Ref
- [8] . 2020. ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery 34, 5 (
Sept. 2020), 1454–1495.DOI: Google ScholarDigital Library
- [9] . 2021. MiniRocket: A very fast (almost) deterministic transform for time series classification. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (
KDD’21 ). Association for Computing Machinery, New York, NY, 248–257.DOI: Google ScholarDigital Library
- [10] . 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7 (2006), 1–30.Google Scholar
Digital Library
- [11] . 2018. Convolutional social pooling for vehicle trajectory prediction. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops’18). IEEE Computer Society, 1468–1476.
DOI: Google ScholarCross Ref
- [12] . 1940. A comparison of alternative tests of significance for the problem of m rankings. Annals of Mathematical Statistics 11, 1 (1940), 86–92.Google Scholar
Cross Ref
- [13] . 2017. Convolutional sequence to sequence learning. CoRR abs/1705.03122 (2017). arXiv:1705.03122 http://arxiv.org/abs/1705.03122.Google Scholar
- [14] . 2018. Towards understanding ECG rhythm classification using convolutional neural networks and attention mappings. In
Proceedings of Machine Learning Research , , , , , , , and (Eds.), Vol. 85. PMLR, Palo Alto, CA, 83–101. http://proceedings.mlr.press/v85/goodfellow18a.html.Google Scholar - [15] . 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In International Conference on Learning Representations (ICLR’16).Google Scholar
- [16] . 2015. Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). arXiv:1512.03385 http://arxiv.org/abs/1512.03385.Google Scholar
- [17] . 2017. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).Google Scholar
Cross Ref
- [18] . 2014. Distilling the knowledge in a neural network. InNIPS abs/1503.02531 (2014).Google Scholar
- [19] . 1997. Long short-term memory. Neural Computing 9, 8 (
Nov. 1997), 1735–1780.DOI: Google ScholarDigital Library
- [20] . 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65–70. http://www.jstor.org/stable/4615733.Google Scholar
- [21] . 2019. Deep learning for time series classification: A review. Data Mining and Knowledge Discovery 33, 4 (2019), 917–963.Google Scholar
Digital Library
- [22] . 2020. InceptionTime: Finding AlexNet for time series classification. Data Mining and Knowledge Discovery 34, 6 (
Nov. 2020), 1936–1962.DOI: Google ScholarDigital Library
- [23] . 2021. An empirical survey of data augmentation for time series classification with neural networks. PLOS ONE 16, 7 (
July 2021), e0254841.DOI: Google ScholarCross Ref
- [24] . 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google Scholar
Cross Ref
- [25] . 2016. Data augmentation for time series classification using convolutional neural networks. In ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data.Google Scholar
- [26] . 2017. Temporal convolutional networks for action segmentation and detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 1003–1012.Google Scholar
Cross Ref
- [27] . 2019. Implementing neural machine translation with bi-directional GRU and attention mechanism on FPGAs using HLS. In Proceedings of the 24th Asia and South Pacific Design Automation Conference (ASPDAC’19). Association for Computing Machinery, New York, NY, 693–698.
DOI: Google ScholarDigital Library
- [28] . 2020. Standalone systolic profile detection of non-contact SCG signal with LSTM network. IEEE Sensors Journal 20, 6 (2020), 3123–3131.
DOI: Google ScholarCross Ref
- [29] . 2016. HIVE-COTE: The hierarchical vote collective of transformation-based ensembles for time series classification. In 2016 IEEE 16th International Conference on Data Mining (ICDM’16). 1041–1046.
DOI: Google ScholarCross Ref
- [30] . 2020. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA’20). IEEE, 9638–9644.
DOI: Google ScholarCross Ref
- [31] . 2017. Pruning convolutional neural networks for resource efficient transfer learning. ICLR 14, 192 (2017), 192.Google Scholar
- [32] . 2021. Convolutional neural network with an elastic matching mechanism for time series classification. Algorithms 14, 7 (2021), 6875–6879.
DOI: Google ScholarCross Ref
- [33] . 2019. TCNN: Temporal convolutional neural network for real-time speech enhancement in the time domain. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’19). 6875–6879.
DOI: Google ScholarCross Ref
- [34] . 2020. LSTM-based ECG classification for continuous monitoring on personal wearable devices. IEEE Journal of Biomedical and Health Informatics 24, 2 (2020), 515–523.
DOI: Google ScholarCross Ref
- [35] . 2019. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In Advances in Neural Information Processing Systems. 4837–4846.Google Scholar
- [36] . 2017. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI’17). AAAI Press, 4278–4284.Google Scholar
Cross Ref
- [37] . 2017. Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI’17). Association for Computing Machinery, New York, NY, 216–220.
DOI: Google ScholarDigital Library
- [38] . 2016. WaveNet: A generative model for raw audio. ArXiv abs/1609.03499 (2016).Google Scholar
- [39] . 2020. Mcfly: Automated deep learning on time series. SoftwareX 12 (2020), 100548.
DOI: Google ScholarCross Ref
- [40] . 1989. Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing 37, 3 (1989), 328–339.
DOI: Google ScholarCross Ref
- [41] . 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics. Springer, 196–202.Google Scholar
Cross Ref
- [42] . 2021. Congestion-aware multi-agent trajectory prediction for collision avoidance. CoRR abs/2103.14231 (2021). arXiv:2103.14231 https://arxiv.org/abs/2103.14231.Google Scholar
- [43] . 2020. Neural machine translation with GRU-gated attention model. IEEE Transactions on Neural Networks and Learning Systems (2020), 1–11.
DOI: Google ScholarCross Ref
- [44] . 2018. Long short-term memory recurrent neural network for remaining useful life prediction of lithium-Ion batteries. IEEE Transactions on Vehicular Technology 67, 7 (2018), 5695–5705.
DOI: Google ScholarCross Ref
- [45] . 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2921–2929.Google Scholar
Cross Ref
Index Terms
ATCN: Resource-efficient Processing of Time Series on Edge
Recommendations
Forecasting Sunspot Numbers with Recurrent Neural Networks (RNN) Using 'Sunspot Neural Forecaster' System
ACT '10: Proceedings of the 2010 Second International Conference on Advances in Computing, Control, and Telecommunication TechnologiesThis paper presents the investigations of forecasting performance of different type of Recurrent Neural Networks (RNN) in forecasting the sunspot numbers. Recurrent Neural Network will be used in this investigation by using different learning algorithms,...
Modeling and Verification of Zhang Neural Networks for Online Solution of Time-Varying Quadratic Minimization and Programming
ISICA '09: Proceedings of the 4th International Symposium on Advances in Computation and IntelligenceIn this paper, by following Zhang <em>et al</em> 's neural-dynamic method proposed formally since March 2001, two recurrent neural networks are generalized to solve online the time-varying convex quadratic-minimization and quadratic-programming (QP) ...
Power Load Forecasting Using a Refined LSTM
ICMLC '19: Proceedings of the 2019 11th International Conference on Machine Learning and ComputingThe power load forecasting is based on historical energy consumption data of a region to forecast the power consumption of the region for a period of time in the future. Accurate forecasting can provide effective and reliable guidance for power ...






Comments