There has been a recent surge of interest in estimation theory based on very simple noise descriptions; the absolute value of a noise sample is simply bounded. To date this line of work has not been critically compared to pre-existing work on stochastic estimation theory which uses more complicated noise descriptions. The present paper attempts to redress this gap by examining the rapproachment between the two schools of work. For example, we show that for many problems a bounded error estimation scheme is precisely equivalent in terms of final result to the stochastic viewpoint of Bayesian estimation. Much of the interest in bounded error estimation theory stems from offers its advantages of being simple and intuitive. However, as we show, it is demanding on the quantitative accuracy of prior information. In contrast, we discuss how the assumptions underlying stochastic estimation theory are more complex, but have a key feature. Qualitative assumptions on the nature of a typical disturbance sequence can be made to reduce the importance on quantitative assumptions being correct. We also discuss how stochastic theory can be extended to deal with a problem at present tackled only by bounded error estimation methods: the quantification of estimation errors arising from the presence of undermodelling.