Abstract
Kernel adaptive filters (KAFs) are non-linear filters which can adapt temporally and have the additional benefit of being computationally efficient through use of the “kernel trick”. In a number of real-world applications, such as channel equalisation, the non-linear mapping provides significant improvements over conventional linear techniques such as the least mean squares (LMS) and recursive least squares (RLS) algorithms. Prior works have focused mainly on the theory and accuracy of KAFs, with little research on their implementations. This article proposes several variants of algorithms based on the kernel normalised least mean squares (KNLMS) algorithm which utilise a delayed model update to minimise dependencies. Subsequently, this work proposes corresponding hardware architectures which utilise this delayed model update to achieve high sample rates and low latency while also providing high modelling accuracy. The resultant delayed KNLMS (DKNLMS) algorithms can achieve clock rates up to 12× higher than the standard KNLMS algorithm, with minimal impact on accuracy and stability. A system implementation achieves 250 GOps/s and a throughput of 187.4 MHz on an Ultra96 board with 1.8× higher throughput than previous state of the art.
- Davide Anguita, Alessandro Ghio, Stefano Pischiutta, and Sandro Ridella. 2007. A hardware-friendly support vector machine for embedded automotive applications. In Proceedings of the 2007 International Joint Conference on Neural Networks (IJCNN’07). IEEE, Los Alamitos, CA, 1360--1364.Google Scholar
Cross Ref
- Nachman Aronszajn. 1950. Theory of reproducing kernels. Transactions of the American Mathematical Society 68, 3 (1950), 337--404.Google Scholar
Cross Ref
- Jonathan Bachrach, Huy Vo, Brian Richards, Yunsup Lee, Andrew Waterman, Rimas Avižienis, John Wawrzynek, and Krste Asanović. 2012. Chisel: Constructing hardware in a scala embedded language. In Proceedings of the 49th Annual Design Automation Conference. ACM, New York, NY, 1216--1225.Google Scholar
Digital Library
- Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Girayes, Avi Mendelson, and Alexander M. Bronstein. 2018. Nice: Noise injection and clamping estimation for neural network quantization. arXiv:1810.00162.Google Scholar
- Chaim Baskin, Eli Schwartz, Evgenii Zheltonozhskii, Natan Liss, Raja Giryes, Alex M. Bronstein, and Avi Mendelson. 2018. UNIQ: Uniform noise injection for non-uniform quantization of neural networks. arXiv:1804.10969.Google Scholar
- James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13 (2012), 281--305.Google Scholar
Digital Library
- Chris M. Bishop. 1995. Training with noise is equivalent to Tikhonov regularization. Neural Computation 7, 1 (1995), 108--116.Google Scholar
Digital Library
- Badong Chen, Nanning Zheng, and Jose C. Principe. 2013. Survival kernel with application to kernel adaptive filtering. In Proceedings of the 2013 International Joint Conference on Neural Networks (IJCNN’13). IEEE, Los Alamitos, CA, 1--6.Google Scholar
- Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning 20, 3 (Sept. 1995), 273--297. DOI:https://doi.org/10.1023/A:1022627411411Google Scholar
Digital Library
- Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems. 3123--3131.Google Scholar
- Feng Ding. 2013. Decomposition based fast least squares algorithm for output error systems. Signal Processing 93, 5 (2013), 1235--1242.Google Scholar
Digital Library
- Feng Ding, Ling Xu, Fuad E. Alsaadi, and Tasawar Hayat. 2018. Iterative parameter identification for pseudo-linear systems with ARMA noise using the filtering technique. IET Control Theory 8 Applications 12, 7 (2018), 892--899.Google Scholar
- Scott C. Douglas, Quanhong Zhu, and Kent F. Smith. 1998. A pipelined LMS adaptive FIR filter architecture without adaptation delay. IEEE Transactions on Signal Processing 46, 3 (1998), 775--779.Google Scholar
Digital Library
- Yaakov Engel, Shie Mannor, and Ron Meir. 2003. The kernel recursive least squares algorithm. IEEE Transactions on Signal Processing 52 (2003), 2275--2285.Google Scholar
Digital Library
- Gerald Estrin. 1960. Organization of computer systems: The fixed plus variable structure computer. Paper presented at the Western Joint IRE-AIEE-ACM Computer Conference. ACM, New York, NY, 33--40.Google Scholar
- Sean Fox, David Boland, and Philip Leong. 2018. FPGA Fastfood—A high speed systolic implementation of a large scale online kernel method. In Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA’18). ACM, New York, NY, 279--284. DOI:https://doi.org/10.1145/3174243.3174271Google Scholar
Digital Library
- Sean Fox, Stephen Tridgell, Craig T. Jin, and Philip H. W. Leong. 2016. Random projections for scaling machine learning on FPGAs. In Proceedings of the International Conference on Field Programmable Technology (FPT’16). 85--92. DOI:https://doi.org/10.1109/FPT.2016.7929193Google Scholar
- Nicholas J. Fraser, Junkyu Lee, Duncan J. M. Moss, Julian Faraone, Stephen Tridgell, Craig T. Jin, and Philip H. W. Leong. 2017. FPGA implementations of kernel normalised least mean squares processors. ACM Transactions on Reconfigurable Technology and Systems 10, 4 (Dec. 2017), Article 26, 20 pages. DOI:https://doi.org/10.1145/3106744Google Scholar
Digital Library
- Nicholas J. Fraser, Duncan J. M. Moss, JunKyu Lee, Stephen Tridgell, Craig T. Jin, and Philip H. W. Leong. 2015. A fully pipelined kernel normalised least mean squares processor for accelerated parameter optimisation. In Proceedings of the 2015 25th International Conference on Field Programmable Logic and Applications (FPL’15). IEEE, Los Alamitos, CA, 1--6.Google Scholar
- Barbara Hammer and Kai Gersmann. 2003. A note on the universal approximation capability of support vector machines. Neural Processing Letters 17, 1 (2003), 43--53.Google Scholar
Digital Library
- Simon S. Haykin. 2005. Adaptive Filter Theory. Pearson Education India.Google Scholar
- Jyrki Kivinen, Alexander J. Smola, and Robert C. Williamson. 2004. Online learning with kernels. IEEE Transactions on Signal Processing 52, 8 (2004), 2165--2176.Google Scholar
Digital Library
- Quoc Le, Tamás Sarlós, and Alex Smola. 2013. Fastfood—Approximating kernel expansions in loglinear time. In Proceedings of the International Conference on Machine Learning, Vol. 85.Google Scholar
- Weifeng Liu, José C. Príncipe, and Simon Haykin. 2011. Kernel Adaptive Filtering: A Comprehensive Introduction. Vol. 57. John Wiley 8 Sons.Google Scholar
- Guoz-Hu Long, Fuyun Ling, and John G. Proakis. 1989. The LMS algorithm with delayed coefficient adaptation. IEEE Transactions on Acoustics, Speech and Signal Processing 37, 9 (1989), 1397--1405.Google Scholar
Cross Ref
- Michael C. Mackey and Leon Glass. 1977. Oscillation and chaos in physiological control systems. Science 197, 4300 (1977), 287--289.Google Scholar
Cross Ref
- Yeyong Pang, Shaojun Wang, Yu Peng, Xiyuan Peng, Nicholas J. Fraser, and Philip H. W. Leong. 2016. A microcoded kernel recursive least squares processor using FPGA technology. ACM Transactions on Reconfigurable Technology and Systems 10, 1 (Sept. 2016), Article 5, 22 pages. DOI:https://doi.org/10.1145/2950061Google Scholar
Digital Library
- Rainer D. Poltmann. 1995. Conversion of the delayed LMS algorithm into the LMS algorithm. IEEE Signal Processing Letters 2, 12 (1995), 223.Google Scholar
Cross Ref
- Xiaowei Ren, Pengju Ren, Badong Chen, Tai Min, and Nanning Zheng. 2014. Hardware implementation of KLMS algorithm using FPGA. In Proceedings of the 2014 International Joint Conference on Neural Networks (IJCNN’14). IEEE, Los Alamitos, CA, 2276--2281.Google Scholar
Cross Ref
- Cédric Richard, José Carlos M. Bermudez, and Paul Honeine. 2009. Online prediction of time series data with kernels. IEEE Transactions on Signal Processing 57, 3 (2009), 1058--1067.Google Scholar
Digital Library
- Sherif A. Tawfik. 2005. Minimax Approximation and Remez Algorithm. Retrieved January 24, 2020 from http://www.math.unipd.it/∼alvise/CS_2008/APPROSSIMAZIONE_2009/MFILES/Remez.pdf.Google Scholar
- Stephen Tridgell, Duncan J. M. Moss, Nicholas J. Fraser, and Philip H. W. Leong. 2015. Braiding: A scheme for resolving hazards in kernel adaptive filters. In Proceedings of the International Conference on Field Programmable Technology (FPT’15). 136--143. DOI:https://doi.org/10.1109/FPT.2015.7393140Google Scholar
- Steven Van Vaerenbergh. 2012. Kernel Methods Toolbox KAFBOX: A Matlab Benchmarking Toolbox for Kernel Adaptive Filtering. Software available at http://sourceforge.net/p/kafbox.Google Scholar
- Steven Van Vaerenbergh, I. Santamaria, Weifeng Liu, and J. C. Príncipe. 2010. Fixed-budget kernel recursive least-squares. In Proceedings of the 2010 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’10). IEEE, Los Alamitos, CA, 1882--1885.Google Scholar
- S. Van Vaerenbergh, J. Via, and I. Santamaria. 2006. A sliding-window kernel RLS algorithm and its application to nonlinear channel identification. In Proceedings of the 2006 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’06), Vol. 5. DOI:https://doi.org/10.1109/ICASSP.2006.1661394Google Scholar
Cross Ref
- Qian Wang, Xianyi Zhang, Yunquan Zhang, and Qing Yi. 2013. AUGEM: Automatically generate high performance dense linear algebra kernels on x86 CPUs. In Proceedings of the 2013 International Conference on High Performance Computing, Networking, Storage, and Analysis (SC’13). IEEE, Los Alamitos, CA, 1--12.Google Scholar
Digital Library
- R. Clint Whaley and Antoine Petitet. 2005. Minimizing development and maintenance costs in supporting persistently optimized BLAS. Software: Practice and Experience 35, 2 (Feb. 2005), 101--121.Google Scholar
- B. Widrow and M. E. Hoff Jr. 1960. Adaptive switching circuits. In IRE WESCON Convention Record. 96--104.Google Scholar
- Ying Yi, Roger Woods, Lok-Kee Ting, and C. F. N. Cowan. 2005. High speed FPGA-based implementations of delayed-LMS filters. Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology 39, 1--2 (2005), 113--131.Google Scholar
Digital Library
Index Terms
Kernel Normalised Least Mean Squares with Delayed Model Adaptation
Recommendations
Total least mean squares algorithm
Widrow (1971) proposed the least mean squares (LMS) algorithm, which has been extensively applied in adaptive signal processing and adaptive control. The LMS algorithm is based on the minimum mean squares error. On the basis of the total least mean ...
Kernel least mean square with adaptive kernel size
Kernel adaptive filters (KAF) are a class of powerful nonlinear filters developed in Reproducing Kernel Hilbert Space (RKHS). The Gaussian kernel is usually the default kernel in KAF algorithms, but selecting the proper kernel size (bandwidth) is still ...
Least squares kernel ensemble regression in Reproducing Kernel Hilbert Space
AbstractEnsemble regression method shows better performance than single regression since ensemble regression method can combine several single regression methods together to improve accuracy and stability of a single regressor. In this paper, ...






Comments