How should one compute a multiple linear regression, which amounts to solving a symmetric system of...
analysis/multiple.regression New
New entry, edited from material collected for the archive and not yet in a released version.
Problem
How should one compute a multiple linear regression, which amounts to solving a symmetric system of simultaneous linear equations? I have another business math question. (The last one was about how you compute an IRR which is a 400-degree polynomial.) This week's job is to do multiple linear regression, a statistical technique which can generate invalid conclusions faster than about any other. Multiple regression involves solving a bunch of simultaneous linear equations, which is equivalent to inverting a matrix. The matrix turns out to be symmetric around the diagonal, which makes life somewhat easier. Before I plunge into this, I'd appreciate hearing from anybody else with experience in doing this. Actual code would particularly be appreciated. -- John R. Levine <johnl@ima.UUCP> [ Editor: first sentence of the problem statement written by the editor from the posting. ]
Show solution
The exact techniques used for multi-linreg are many and varied. The most accurate method (and most expensive) is single point decomposition. Alternatives include Gram-Schmidt orthogonalization, LU decomposition, Cholesky factorization, etc. etc. Which method one chooses depends in part upon the size of the model, in part on available memory (i.e. can the matrices fit or are sparse matrix techniques required), how much auto-correlation there is among the variables, etc. etc. I worked for several years doing econometrics programming, working for Data Resources Inc. in Lexington Mass. The sheer volume of code that we used in implementing regression algorithms was staggering. We had to handle everything from simple 2-3 variable regressions to large (100's) models of the U.S. economy. As a good 'overall' algorithm which compromises well between space and speed I would use a [packed] Cholesky factorization. One must first of course estimate the condition numbers of the matrix (which is positive definite) before embarking on doing the factorization. There are a large number of algorithms in the collected algorithms of the ACM and an even better source is the LINPACK book by Jack Dongarra et.al. -- Bob Silverman [ Editor: "single point decomposition" is presumably singular value decomposition (SVD), which is the method usually described this way. ]