Symbolic Regression Based Feature Extraction of Shallow Neural-Networks for Identification and Prediction

Selami Beyhan

¹ Izmir Demokrasi University, Electrical and Electronics Engineering, Uckuyular Dist., Karabaglar 35140, Izmir, Turkey

^* Corresponding author’s E-mail: [email protected]

This paper proposes a feature extraction method to improve the performance of shallow neural-network models with less number of parameters to apply especially on embedded system design at remote applications. Feature extraction method is designed using fuzzy c-means clustering based fuzzy system design cascaded a layer of symbolic operators and functions, respectively. During the training stage of neural-networks, symbolic operators and functions are selected using random-learning theory with the unity internal weights such that based on the prediction performance, optimal sequences are recorded for feature extraction to be utilized on testing phase. Extracted features are here used to empower the single-layer neural-network (SLNN) with sigmoid hyperbolic activation functions, functional-link neural- network (FLNN) with Chebyshev polynomials and Pi-Sigma higher-order neural-network (PSNN) with sigmoid activation functions, respectively. The internal and output parameters of the appended shallow neural-networks are optimized using batch optimization methods. Proposed regression models are first tested on identification of an artificial discrete-time dynamic system and real-time inverted pendulum then also for prediction of the sunspot time-series and traffic density estimation. As a result, the prediction performance of shallow neural networks is improved to be used in future applications.

Symbolic regression, clustering, shallow neural-network, time-series prediction, system identification

Selami Beyhan (2022). Symbolic Regression Based Feature Extraction of Shallow Neural-Networks for Identification and Prediction. Journal of Artificial Intelligence and Systems, 4, 33–49. https://doi.org/10.33969/AIS.2022040103.

[1] S. S. Alahmari, D. B. Goldgof, P. R. Mouton, and L. O. Hall, “Challenges for the repeatability of deep learning models,” IEEE Access, vol. 8, pp. 211 860–211 868, 2020.

[2] X. Feng, Y. Jiang, X. Yang, M. Du, and X. Li, “Computer vision algorithms and hardware implementations: A survey,” Integration, vol. 69, pp. 309–320, 2019.

[3] A. Ananthakrishnan and M. G. Allen, “All-passive hardware implementation of multilayer perceptron classifiers,” IEEE Transactions on Neural Networks and Learning Systems, 2020.

[4] H. Younes, A. Ibrahim, M. Rizk, and M. Valle, “Efficient fpga implementation of approximate singular value decomposition based on shallow neural networks,” in 2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS).    IEEE, 2021, pp. 1–4.

[5] J. Yan, X. Guo, S. Duan, P. Jia, L. Wang, C. Peng, and S. Zhang, “Electronic nose feature extraction methods: A review,” Sensors, vol. 15, no. 11, pp. 27 804–27 831, 2015.

[6] S. Sachar and A. Kumar, “Survey of feature extraction and classification techniques to identify plant through leaves,” Expert Systems with Applications, vol. 167, p. 114181, 2021.

[7] J. R. Koza and J. R. Koza, Genetic programming: on the programming of computers by means of natural selection.    MIT press, 1992, vol. 1.

[8] J. Mada´r, J. Abonyi, and F. Szeifert, “Genetic programming for the identification of nonlinear input- output models,” Industrial & engineering chemistry research, vol. 44, no. 9, pp. 3178–3186, 2005.

[9] J. Kubal´ık, E.  Derner,  and R. Babusˇka,  “Multi-objective symbolic regression for physics-aware dynamic modeling,” Expert Systems with Applications, vol. 182, p. 115210, 2021.

[10] A. Diveev, S. Konstantinov, and A. Danilova, “Solution of the optimal control problem by symbolic regression method,” Procedia Computer Science, vol. 186, pp. 646–653, 2021.

[11] G. H. Yamashita, F. S. Fogliatto, M. J. Anzanello, and G. L. Tortorella, “Customized prediction of attendance to soccer matches based on symbolic regression and genetic programming,” Expert Systems with Applications, vol. 187, p. 115912, 2022.

[12] B. E. Ko¨ ktu¨ rk-Gu¨ zel and S. Beyhan, “Symbolic regression based extreme learning machine models for system identification,” Neural Processing Letters, vol. 53, no. 2, pp. 1565–1578, 2021.

[13] F. O. de Franca and M. Z. de Lima, “Interaction-transformation symbolic regression with extreme learning machine,” Neurocomputing, vol. 423, pp. 609–619, 2021.

[14] S. Beyhan, “Fuzzy emulated symbolic regression for modelling and control of markov jump systems with unknown transition rates,” IEEE Transactions on Circuits and Systems II: Express Briefs, 2021.

[15] Y. Shin and J. Ghosh, “The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation,” in IJCNN-91-Seattle international joint conference on neural networks, vol. 1.   IEEE, 1991, pp. 13–18.

[16] E. Egrioglu and E. Bas, “Modified pi sigma artificial neural networks for forecasting,” Granular Computing, pp. 1–5, 2022.

[17] S. Beyhan and M. Alci, “Fuzzy functions based arx model and new fuzzy basis function models for nonlinear system identification,” Applied Soft Computing, vol. 10, no. 2, pp. 439–444, 2010.

[18] M. Alci and M. H. Asyali, “Nonlinear system identification via laguerre network based fuzzy systems,” Fuzzy sets and systems, vol. 160, no. 24, pp. 3518–3529, 2009.

[19] M. no. 33-936S Ed01 122006, “Manual on digital pendulum control experiments,” 2011.

[20] Y. Xixi, D. Fengqian, and L. Chao, “Time series prediction based on high-order intuitionistic fuzzy cognitive maps with variational mode decomposition,” Soft Computing, vol. 26, no. 1, pp. 189–201, 2022.