Linear progression is a type of programming where weight is added from workout to workout as opposed to on a weekly basis (intermediate level programming), or on even longer timescales (advanced periodization concepts).
This is possible during the novice phase only, because the novice's ability to inflict stress via a workout has not surpassed their ability to recover from that stress. In other words, they recover completely from Workout to workout. Put another way, every subsequent Workout represents a new peak.
Contrast that to the intermediate, who has to be content with week to week improvement. For example, in the Texas Method, you try to increase last Monday's 5x5 load on the following Monday, not on the Wednesday 'light' day or the Friday 'intensity' day.
To answer the question as to why linear progression is so effective: it's simply the fastest way to increase load. Why wait a week to add 5lbs when you can do it on the next Workout?
Consider two rank beginners, one on a beginner program and one on an intermediate. They both squat 100 pounds on day 1, a Monday. By the following Monday, one beginner using linear progression has added at least 10 pounds in the meantime and begins the workout at 115 lbs. The other beginner is held back by his own program and begins the second Monday Workout at 105. Follow this logic out a few weeks or months, and the beginner using linear progression is miles ahead. **I'm assuming both take the advice to add weight in the smallest possible increment.**
Only when you can no longer recover from Workout to workout, i.e. when you can't make all the sets and reps with the new load, do you consider more advanced programming. Even then, you can stretch the progression out a bit longer by deloading and working back up to the weight. This trick seems to work once, or twice if you're lucky.
*I'm using the terms novice/beginner, intermediate, and advanced as explained in Practical Programming. Other authors have perfectly valid definitions for these terms, but I think Kilgore's are the most useful when discussing programming.