Linear Regression Implementation

Implement linear regression from scratch using gradient descent.

Given:

• Training data: X (n × m matrix) and y (n × 1 vector)
• X contains n samples with m features
• y contains n target values

Implement a LinearRegression class with:

• fit(X, y, learning_rate, num_iterations) - train the model
• predict(X) - make predictions
• Calculate and use gradient descent to optimize weights

Example:

# Training data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([2, 4, 6, 8, 10])

# Train model
model = LinearRegression()
model.fit(X, y, learning_rate=0.01, num_iterations=1000)

# Make predictions
predictions = model.predict(np.array([[6], [7]]))
# Expected: [12, 14] (approximately)

Hints

Hint 1

Linear regression follows the formula: ŷ = Xw + b, where w is weights and b is bias. How do you initialize these parameters?

Hint 2

The loss function is Mean Squared Error: L = (1/n) × Σ(yi - ŷi)². You need to compute gradients of this with respect to w and b.

Hint 3

Gradient descent update rule: w = w - α × (∂L/∂w), where α is learning rate. The gradient ∂L/∂w = -(2/n) × X^T × (y - ŷ).