A Bayesian approach to artificial neural network model selection

© 2005 Modelling & Simulation Society of Australia & New Zealand

Artificial neural networks (ANNs) have proven to be extremely valuable tools in the field of water resources engineering. However, one of the most difficult tasks in developing an ANN is determining the optimum level of complexity required to model a given problem, as there is no formal systematic model selection method. The generalisability of an ANN, which is defined by its predictive performance on the universe of possible data, can be significantly impaired if there are too few or too many hidden nodes in the network. Therefore, for an ANN to be a valuable prediction tool, it is important that some effort is made to optimise the number of hidden nodes. This paper presents a Bayesian model selection (BMS) method for ANNs that provides an objective approach for comparing models of varying complexity in order to select the most appropriate ANN structure. Given a set of competing models ℋ1, . . . ,ℋH, BMS is used to compare the posterior probability that each model ℋi is the true data generating function, given a set of observed data y. This probability is also known as the evidence of a model and the ratio of two competing models' evidence values, known as the Bayes' factor, can be used to rank the competing models in terms of the relative evidence in support of each model. For ANNs (and other complex models), the evidence of a model p(ℋ|y) is analytically intractable and, consequently, alternative methods are required to estimate these probabilities for the competing models. One such method involves the use of Markov chain Monte Carlo (MCMC) simulations from the posterior weight distribution p(w|y, ℋ) to approximate the evidence. It has already been shown that there are numerous benefits to estimating the posterior distribution of ANN weights with MCMC methods; therefore, the proposed BMS approach is based on such an approximation of p(y|ℋ), as this only requires a simple additional step after sampling from p(w|y, ℋ). Furthermore, the weight distributions obtained from the MCMC simulation provide a useful check of the accuracy to the approximated Bayes' factors. A problem associated with the use of posterior simulations to estimate a model's evidence is that the approximation may be sensitive to factors associated with the MCMC simulation. Therefore, the proposed BMS method for ANNs incorporates a further check of the accuracy of the computed Bayes' factors by inspecting the marginal posterior distributions of the hidden-to-output layer weights, which indicate whether all of the hidden nodes in the model are necessary. The fact that this check is available is one of the greatest advantages of the proposed approach over conventional model selection methods, which do not provide such a test and instead rely on the modeller's subjective choice of selection criterion. The aim of model selection is to enable generalisation to new cases. Therefore, in the case study presented in this paper, the performance of the proposed BMS method was assessed in comparison to the performance of conventional ANN selection methods on data outside the domain of the training data. This case study, which involves forecasting salinity concentrations in the River Murray at Murray Bridge, South Australia, 14 days in advance, was chosen as it had been shown previously that, if an ANN was trained on the first half of the available data, it would be required to extrapolate in some cases when applied to the second half of the available data set. In this case study, the proposed BMS framework for ANNs was shown to be more successful than conventional model selection methods in selecting an ANN that could approximate the relationship contained in the training data and generalise to new cases outside the domain of those used for training. The Bayes' factors calculated were useful for obtaining an initial guide to the most appropriate model; however, the final step involving inspection of marginal posterior hidden-tooutput weight distributions was necessary for the final selection of the optimum number of hidden nodes. The model selected using the proposed BMS approach not only had the best generalisability, but was also more parsimonious than the models selected using conventional methods and required considerably less time for training.

Kingston, G. B., H. R. Maier and M. F. Lambert

http://www.mssanz.org.au/modsim05/index.htm