Extension to Multiple Features

Linear regression can be extended to include multiple features, allowing us to model more complex relationships.

The Model

With multiple features, our model becomes:

$f(\mathbf{x}) = \beta_0 + \sum_{j=1}^d \beta_jx_j$

Where:

• $d$ is the number of features
• $\mathbf{x}$ is a vector of feature values $(x_1, x_2, ..., x_d)$
• $\beta_j$ are the coefficients (including $\beta_0$ for the intercept)

Example

Let's extend our study hours vs. test scores example to include two features:

1. Study hours: $x_1$
2. Previous test scores: $x_2$

Our model would now look like:

$\text{Predicted Score} = \beta_0 + \beta_1(\text{Study Hours}) + \beta_2(\text{Previous Score})$

Gradient Descent for Multiple Features

The gradient descent algorithm can be generalized for multiple features:

Repeat until convergence: Update simultaneously all $\beta_j$ for $(j = 0, 1, ..., d)$

$\beta_j := \beta_j - \alpha \frac{\partial}{\partial \beta_j} R(\beta)$

Where:

$\frac{\partial R}{\partial \beta_j} = \frac{1}{n} \sum_{i=1}^n (f(\mathbf{x}^{(i)}) - y^{(i)}) x_j^{(i)}$

• $f(\mathbf{x}^{(i)})$ is our prediction for the $i-$th example
• $x_j^{(i)}$ is the $j-$ th feature of the $i-$ th example (define $x_0^{(i)} = 1$ for all $i$)

Practical Considerations

1. Feature scaling becomes even more important with multiple features to ensure all features contribute proportionally to the model.
2. Be cautious of multicollinearity, where features are highly correlated with each other, which can make the model unstable.
3. As the number of features increases, the risk of overfitting also increases. Consider using regularization techniques like Ridge or Lasso regression.

By extending to multiple features, we can create more sophisticated models that capture complex relationships in our data, potentially leading to more accurate predictions.