 Machine Learning
Polynomial Regression in Machine Learning: A Beginner Tutorial

9 A quadratic equation is of the form y = ax2 + bx + c.

What if your data is actually more complex than a simple straight line? Surprisingly,

you can actually use a linear model to fit nonlinear data. A simple way to do this is to

add powers of each feature as new features, then train a linear model on this extended

set of features. This technique is called Polynomial Regression.

Let’s look at an example. First, let’s generate some nonlinear data, based on a simple

quadratic equation9 (plus some noise; see Figure 4-12):

m = 100

X = 6 * np.random.rand(m, 1) - 3

y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)

Figure 4-12. Generated nonlinear and noisy dataset

nomialFeatures class to transform our training data, adding the square (2nd-degree

polynomial) of each feature in the training set as new features (in this case there is

just one feature):

>>> from sklearn.preprocessing import PolynomialFeatures

>>> poly_features = PolynomialFeatures(degree=2, include_bias=False)

>>> X_poly = poly_features.fit_transform(X)

>>> X

array([-0.75275929])

>>> X_poly

array([-0.75275929, 0.56664654])

X_poly now contains the original feature of X plus the square of this feature. Now you

can fit a LinearRegression model to this extended training data (Figure 4-13):

>>> lin_reg = LinearRegression()

>>> lin_reg.fit(X_poly, y)

>>> lin_reg.intercept_, lin_reg.coef_

(array([ 1.78134581]), array([[ 0.93366893, 0.56456263]]))

Figure 4-13. Polynomial Regression model predictions

Not bad: the model estimates y = 0 . 56x1

2 + 0 . 93x1 + 1 . 78 when in fact the original

function was y = 0 . 5x1

2 + 1 . 0x1 + 2 . 0 + Gaussian noise.

Note that when there are multiple features, Polynomial Regression is capable of find‐

ing relationships between features (which is something a plain Linear Regression

model cannot do). This is made possible by the fact that PolynomialFeatures also

adds all combinations of features up to the given degree. For example, if there were

features a2, a3, b2, and b3, but also the combinations ab, a2b, and ab2.

PolynomialFeatures(degree=d) transforms an array containing n

features into an array containing n + d !

d! n! features, where n! is the

factorial of n, equal to 1 × 2 × 3 × × n. Beware of the combinato‐

rial explosion of the number of features!