skip to main content
research-article

Kernel Normalised Least Mean Squares with Delayed Model Adaptation

Published:13 February 2020Publication History
Skip Abstract Section

Abstract

Kernel adaptive filters (KAFs) are non-linear filters which can adapt temporally and have the additional benefit of being computationally efficient through use of the “kernel trick”. In a number of real-world applications, such as channel equalisation, the non-linear mapping provides significant improvements over conventional linear techniques such as the least mean squares (LMS) and recursive least squares (RLS) algorithms. Prior works have focused mainly on the theory and accuracy of KAFs, with little research on their implementations. This article proposes several variants of algorithms based on the kernel normalised least mean squares (KNLMS) algorithm which utilise a delayed model update to minimise dependencies. Subsequently, this work proposes corresponding hardware architectures which utilise this delayed model update to achieve high sample rates and low latency while also providing high modelling accuracy. The resultant delayed KNLMS (DKNLMS) algorithms can achieve clock rates up to 12× higher than the standard KNLMS algorithm, with minimal impact on accuracy and stability. A system implementation achieves 250 GOps/s and a throughput of 187.4 MHz on an Ultra96 board with 1.8× higher throughput than previous state of the art.

References

  1. Davide Anguita, Alessandro Ghio, Stefano Pischiutta, and Sandro Ridella. 2007. A hardware-friendly support vector machine for embedded automotive applications. In Proceedings of the 2007 International Joint Conference on Neural Networks (IJCNN’07). IEEE, Los Alamitos, CA, 1360--1364.Google ScholarGoogle ScholarCross RefCross Ref
  2. Nachman Aronszajn. 1950. Theory of reproducing kernels. Transactions of the American Mathematical Society 68, 3 (1950), 337--404.Google ScholarGoogle ScholarCross RefCross Ref
  3. Jonathan Bachrach, Huy Vo, Brian Richards, Yunsup Lee, Andrew Waterman, Rimas Avižienis, John Wawrzynek, and Krste Asanović. 2012. Chisel: Constructing hardware in a scala embedded language. In Proceedings of the 49th Annual Design Automation Conference. ACM, New York, NY, 1216--1225.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Girayes, Avi Mendelson, and Alexander M. Bronstein. 2018. Nice: Noise injection and clamping estimation for neural network quantization. arXiv:1810.00162.Google ScholarGoogle Scholar
  5. Chaim Baskin, Eli Schwartz, Evgenii Zheltonozhskii, Natan Liss, Raja Giryes, Alex M. Bronstein, and Avi Mendelson. 2018. UNIQ: Uniform noise injection for non-uniform quantization of neural networks. arXiv:1804.10969.Google ScholarGoogle Scholar
  6. James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13 (2012), 281--305.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Chris M. Bishop. 1995. Training with noise is equivalent to Tikhonov regularization. Neural Computation 7, 1 (1995), 108--116.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Badong Chen, Nanning Zheng, and Jose C. Principe. 2013. Survival kernel with application to kernel adaptive filtering. In Proceedings of the 2013 International Joint Conference on Neural Networks (IJCNN’13). IEEE, Los Alamitos, CA, 1--6.Google ScholarGoogle Scholar
  9. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning 20, 3 (Sept. 1995), 273--297. DOI:https://doi.org/10.1023/A:1022627411411Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems. 3123--3131.Google ScholarGoogle Scholar
  11. Feng Ding. 2013. Decomposition based fast least squares algorithm for output error systems. Signal Processing 93, 5 (2013), 1235--1242.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Feng Ding, Ling Xu, Fuad E. Alsaadi, and Tasawar Hayat. 2018. Iterative parameter identification for pseudo-linear systems with ARMA noise using the filtering technique. IET Control Theory 8 Applications 12, 7 (2018), 892--899.Google ScholarGoogle Scholar
  13. Scott C. Douglas, Quanhong Zhu, and Kent F. Smith. 1998. A pipelined LMS adaptive FIR filter architecture without adaptation delay. IEEE Transactions on Signal Processing 46, 3 (1998), 775--779.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Yaakov Engel, Shie Mannor, and Ron Meir. 2003. The kernel recursive least squares algorithm. IEEE Transactions on Signal Processing 52 (2003), 2275--2285.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Gerald Estrin. 1960. Organization of computer systems: The fixed plus variable structure computer. Paper presented at the Western Joint IRE-AIEE-ACM Computer Conference. ACM, New York, NY, 33--40.Google ScholarGoogle Scholar
  16. Sean Fox, David Boland, and Philip Leong. 2018. FPGA Fastfood—A high speed systolic implementation of a large scale online kernel method. In Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA’18). ACM, New York, NY, 279--284. DOI:https://doi.org/10.1145/3174243.3174271Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Sean Fox, Stephen Tridgell, Craig T. Jin, and Philip H. W. Leong. 2016. Random projections for scaling machine learning on FPGAs. In Proceedings of the International Conference on Field Programmable Technology (FPT’16). 85--92. DOI:https://doi.org/10.1109/FPT.2016.7929193Google ScholarGoogle Scholar
  18. Nicholas J. Fraser, Junkyu Lee, Duncan J. M. Moss, Julian Faraone, Stephen Tridgell, Craig T. Jin, and Philip H. W. Leong. 2017. FPGA implementations of kernel normalised least mean squares processors. ACM Transactions on Reconfigurable Technology and Systems 10, 4 (Dec. 2017), Article 26, 20 pages. DOI:https://doi.org/10.1145/3106744Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Nicholas J. Fraser, Duncan J. M. Moss, JunKyu Lee, Stephen Tridgell, Craig T. Jin, and Philip H. W. Leong. 2015. A fully pipelined kernel normalised least mean squares processor for accelerated parameter optimisation. In Proceedings of the 2015 25th International Conference on Field Programmable Logic and Applications (FPL’15). IEEE, Los Alamitos, CA, 1--6.Google ScholarGoogle Scholar
  20. Barbara Hammer and Kai Gersmann. 2003. A note on the universal approximation capability of support vector machines. Neural Processing Letters 17, 1 (2003), 43--53.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Simon S. Haykin. 2005. Adaptive Filter Theory. Pearson Education India.Google ScholarGoogle Scholar
  22. Jyrki Kivinen, Alexander J. Smola, and Robert C. Williamson. 2004. Online learning with kernels. IEEE Transactions on Signal Processing 52, 8 (2004), 2165--2176.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Quoc Le, Tamás Sarlós, and Alex Smola. 2013. Fastfood—Approximating kernel expansions in loglinear time. In Proceedings of the International Conference on Machine Learning, Vol. 85.Google ScholarGoogle Scholar
  24. Weifeng Liu, José C. Príncipe, and Simon Haykin. 2011. Kernel Adaptive Filtering: A Comprehensive Introduction. Vol. 57. John Wiley 8 Sons.Google ScholarGoogle Scholar
  25. Guoz-Hu Long, Fuyun Ling, and John G. Proakis. 1989. The LMS algorithm with delayed coefficient adaptation. IEEE Transactions on Acoustics, Speech and Signal Processing 37, 9 (1989), 1397--1405.Google ScholarGoogle ScholarCross RefCross Ref
  26. Michael C. Mackey and Leon Glass. 1977. Oscillation and chaos in physiological control systems. Science 197, 4300 (1977), 287--289.Google ScholarGoogle ScholarCross RefCross Ref
  27. Yeyong Pang, Shaojun Wang, Yu Peng, Xiyuan Peng, Nicholas J. Fraser, and Philip H. W. Leong. 2016. A microcoded kernel recursive least squares processor using FPGA technology. ACM Transactions on Reconfigurable Technology and Systems 10, 1 (Sept. 2016), Article 5, 22 pages. DOI:https://doi.org/10.1145/2950061Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Rainer D. Poltmann. 1995. Conversion of the delayed LMS algorithm into the LMS algorithm. IEEE Signal Processing Letters 2, 12 (1995), 223.Google ScholarGoogle ScholarCross RefCross Ref
  29. Xiaowei Ren, Pengju Ren, Badong Chen, Tai Min, and Nanning Zheng. 2014. Hardware implementation of KLMS algorithm using FPGA. In Proceedings of the 2014 International Joint Conference on Neural Networks (IJCNN’14). IEEE, Los Alamitos, CA, 2276--2281.Google ScholarGoogle ScholarCross RefCross Ref
  30. Cédric Richard, José Carlos M. Bermudez, and Paul Honeine. 2009. Online prediction of time series data with kernels. IEEE Transactions on Signal Processing 57, 3 (2009), 1058--1067.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Sherif A. Tawfik. 2005. Minimax Approximation and Remez Algorithm. Retrieved January 24, 2020 from http://www.math.unipd.it/∼alvise/CS_2008/APPROSSIMAZIONE_2009/MFILES/Remez.pdf.Google ScholarGoogle Scholar
  32. Stephen Tridgell, Duncan J. M. Moss, Nicholas J. Fraser, and Philip H. W. Leong. 2015. Braiding: A scheme for resolving hazards in kernel adaptive filters. In Proceedings of the International Conference on Field Programmable Technology (FPT’15). 136--143. DOI:https://doi.org/10.1109/FPT.2015.7393140Google ScholarGoogle Scholar
  33. Steven Van Vaerenbergh. 2012. Kernel Methods Toolbox KAFBOX: A Matlab Benchmarking Toolbox for Kernel Adaptive Filtering. Software available at http://sourceforge.net/p/kafbox.Google ScholarGoogle Scholar
  34. Steven Van Vaerenbergh, I. Santamaria, Weifeng Liu, and J. C. Príncipe. 2010. Fixed-budget kernel recursive least-squares. In Proceedings of the 2010 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’10). IEEE, Los Alamitos, CA, 1882--1885.Google ScholarGoogle Scholar
  35. S. Van Vaerenbergh, J. Via, and I. Santamaria. 2006. A sliding-window kernel RLS algorithm and its application to nonlinear channel identification. In Proceedings of the 2006 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’06), Vol. 5. DOI:https://doi.org/10.1109/ICASSP.2006.1661394Google ScholarGoogle ScholarCross RefCross Ref
  36. Qian Wang, Xianyi Zhang, Yunquan Zhang, and Qing Yi. 2013. AUGEM: Automatically generate high performance dense linear algebra kernels on x86 CPUs. In Proceedings of the 2013 International Conference on High Performance Computing, Networking, Storage, and Analysis (SC’13). IEEE, Los Alamitos, CA, 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. R. Clint Whaley and Antoine Petitet. 2005. Minimizing development and maintenance costs in supporting persistently optimized BLAS. Software: Practice and Experience 35, 2 (Feb. 2005), 101--121.Google ScholarGoogle Scholar
  38. B. Widrow and M. E. Hoff Jr. 1960. Adaptive switching circuits. In IRE WESCON Convention Record. 96--104.Google ScholarGoogle Scholar
  39. Ying Yi, Roger Woods, Lok-Kee Ting, and C. F. N. Cowan. 2005. High speed FPGA-based implementations of delayed-LMS filters. Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology 39, 1--2 (2005), 113--131.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Kernel Normalised Least Mean Squares with Delayed Model Adaptation

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Article Metrics

            • Downloads (Last 12 months)11
            • Downloads (Last 6 weeks)3

            Other Metrics

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!