A quantitative Occam's razor (Q789829)

Given empirical data \(D=\{(x_ i,y_ i): i=1,...,N\}\), the problem is to find a ''theory'' T, i.e. an assignment of a number P(y;x) representing the hypothetical probability of y given x to each pair (x,y)\(\in D\). If - I(T) is the log of the (unnormalized) ''prior probability'' of theory T and \(-I(D| T)=\sum^{N}_{i=1}\log P(y_ i;x_ i)\) is the log probability of D according to T, then \(p(T)=\exp(-I(D| T)-I(T))\) is the ''posterior probability'' of T given data D. Mainly for the nonlinear least-squares regression there is considered that theory T which best fits given data D, i.e. which maximizes p(T).