Support Vector Machine Simplified

We’ll use a tiny dataset with just 4 points. The goal is to separate Class A (+1) and Class B (−1) using a straight line (in 2D).

---

📌 Dataset

Point	x₁	x₂	Class
A	1	2	+1
B	2	3	+1
C	3	3	−1
D	2	1	−1

Plot these on graph paper:

• A (1,2) 🔵

• B (2,3) 🔵

• C (3,3) 🔴

• D (2,1) 🔴

---

✏️ Step 1: Try a line x₂ = x₁ → i.e., line through origin at 45°

The equation of the line is:

$$f(x) = x₂ - x₁ = 0$$

Let’s test each point:

Point	x₁	x₂	f(x) = x₂ - x₁	Result	Prediction
A	1	2	2 - 1 = +1	≥ 0	+1 ✅
B	2	3	3 - 2 = +1	≥ 0	+1 ✅
C	3	3	3 - 3 = 0	≥ 0	+1 ❌
D	2	1	1 - 2 = -1	< 0	-1 ✅

❌ C is wrongly classified. So this line isn’t optimal.

---

✏️ Step 2: Try a better line: x₂ = x₁ + 0.5

This shifts the line upward a bit.
The equation becomes:

$$f(x) = x₂ - x₁ - 0.5 = 0$$

Let’s test:

Point	x₁	x₂	f(x) = x₂ - x₁ - 0.5	Result	Prediction
A	1	2	2 - 1 - 0.5 = +0.5	≥ 0	+1 ✅
B	2	3	3 - 2 - 0.5 = +0.5	≥ 0	+1 ✅
C	3	3	3 - 3 - 0.5 = -0.5	< 0	-1 ✅
D	2	1	1 - 2 - 0.5 = -1.5	< 0	-1 ✅

✅ All 4 points are correctly classified!

---

🧮 Step 3: Express the Equation as SVM Style

SVM wants the line in this form:

$$w₁·x₁ + w₂·x₂ + b = 0$$

Our equation:

$$x₂ - x₁ - 0.5 = 0$$

Can be rewritten as:

$$-x₁ + x₂ - 0.5 = 0$$

So,

• w₁ = -1

• w₂ = +1

• b = -0.5

This is our final separating hyperplane.

---

🧲 Step 4: Margin and Support Vectors

The support vectors are the closest points to the decision boundary.

Check distances from the line:

For point A(1,2):

$$f(x) = 2 - 1 - 0.5 = 0.5$$

Point C(3,3):

$$f(x) = 3 - 3 - 0.5 = -0.5$$

So points A and C are support vectors — they sit at equal distances (margin) from the decision boundary.

---

✅ Final Summary (like a notebook page)

📘 Final Equation:

$$x₂ - x₁ - 0.5 = 0$$

or in SVM form:

$$w = [-1, 1], \quad b = -0.5$$

📍 Support Vectors:

• A(1,2)

• C(3,3)

✅ Classification Rule:

• If f(x) ≥ 0 → Class +1

• If f(x) < 0 → Class -1

---

🎓 SVM in 1 Sentence:

SVM finds the best line (or curve) that maximizes the gap between two classes, using only the closest points (support vectors) to make the decision.

---

---

🎯 GOAL of SVM (in Math Terms)

Given labeled data, find the hyperplane (line) that:

1. Separates the two classes correctly

2. Maximizes the margin (distance from the line to the closest points)

---

✍️ 1. The Equation of a Hyperplane

In 2D, a line is:

$$w_1x_1 + w_2x_2 + b = 0$$

Or, in vector form:

$$\mathbf{w}^\top \mathbf{x} + b = 0$$

• $\mathbf{w} = [w_1, w_2]$ → weight vector (controls the direction of the line)

• $b$ → bias (controls the shift up/down of the line)

• $\mathbf{x} = [x_1, x_2]$ → input point

---

🧠 2. Classification Rule

For any point $\mathbf{x}$:

$$\text{Class} = \begin{cases} +1 & \text{if } \mathbf{w}^\top \mathbf{x} + b \geq 0 \\ -1 & \text{if } \mathbf{w}^\top \mathbf{x} + b < 0 \end{cases}$$

---

📏 3. What is Margin?

Let’s say you have a line that separates the data. The margin is the distance between the line and the closest data points (called support vectors).

We want this margin to be as wide as possible.

Let’s define:

• The distance from a point $\mathbf{x}$ to the line $\mathbf{w}^\top \mathbf{x} + b = 0$ is:

$$\text{Distance} = \frac{|\mathbf{w}^\top \mathbf{x} + b|}{\|\mathbf{w}\|}$$

Where $\|\mathbf{w}\| = \sqrt{w_1^2 + w_2^2}$

---

🏁 4. Optimization Objective

We want:

• All data points classified correctly:

  $$y_i(\mathbf{w}^\top \mathbf{x}_i + b) \geq 1$$

  for all $i$

  This ensures the points are on the correct side of the margin.

• Maximize the margin = Minimize $\|\mathbf{w}\|$

So the optimization problem becomes:

Minimize:

$$\frac{1}{2} \|\mathbf{w}\|^2$$

Subject to:

$$y_i(\mathbf{w}^\top \mathbf{x}_i + b) \geq 1 \quad \text{for all } i$$

This is called a convex optimization problem — it has one global minimum, which we can find using Lagrange Multipliers.

---

🧩 5. Solving Using Lagrangian (Soft Explanation)

We use the method of Lagrange Multipliers to solve this constrained optimization.

We build the Lagrangian:

$$L(\mathbf{w}, b, \boldsymbol{\alpha}) = \frac{1}{2} \|\mathbf{w}\|^2 - \sum_{i=1}^n \alpha_i [y_i(\mathbf{w}^\top \mathbf{x}_i + b) - 1]$$

Where:

• $\alpha_i \geq 0$ are the Lagrange multipliers

Then we find the saddle point (minimize $L$ w.r.t $\mathbf{w}, b$ and maximize w.r.t $\alpha$).

This leads to a dual problem, which is easier to solve using tools like quadratic programming.

---

✳️ 6. Final Classifier

Once solved, we get:

$$\mathbf{w} = \sum_{i=1}^n \alpha_i y_i \mathbf{x}_i$$

This means the support vectors (where $\alpha_i > 0$) are the only ones used to define $\mathbf{w}$. All other data points don’t affect the boundary!

Then you get the decision function:

$$f(\mathbf{x}) = \mathbf{w}^\top \mathbf{x} + b$$

Predict class:

• If $f(\mathbf{x}) \geq 0$ → +1

• If $f(\mathbf{x}) < 0$ → −1

---

🪄 Intuition Summary

Concept	In Simple Words
Hyperplane	The best line that separates classes
Margin	Gap between the line and the nearest points
Support Vectors	Points lying closest to the line
Optimization Goal	Maximize margin (i.e., minimize $\|\mathbf{w}\|$)
Constraint	Keep all points on the correct side
Lagrange Method	A tool to solve optimization with constraints

---