Data Science Multinomial Logistic Regression

🔹 Multinomial Logistic Regression is an extension of Logistic Regression used for multi-class classification (more than two classes).
🔹 Unlike binary logistic regression (which predicts 0 or 1), multinomial logistic regression predicts multiple classes.
🔹 It uses the softmax function instead of the sigmoid function to compute probabilities for each class.

Examples of Multinomial Logistic Regression Applications

✅ Handwritten Digit Recognition (0-9)
✅ Classifying Sentiments (Positive, Neutral, Negative)
✅ Predicting Weather (Sunny, Rainy, Snowy)

2. Mathematical Formula of Multinomial Logistic Regression

Unlike binary logistic regression (which uses the sigmoid function), multinomial logistic regression uses the softmax function:

• \( P(Y=k \mid X) = \frac{e^{m_k X + c_k}}{\sum_{j=1}^{K} e^{m_j X + c_j}} \)
• \( P(Y=k \mid X) \) → Probability that \( Y \) belongs to class \( k \)
• \( m_k \) → Coefficients for class \( k \)
• \( c_k \) → Intercept for class \( k \)
• \( k \) → Total number of classes
• \( e \) → Euler’s number (~2.718)

The model assigns a probability score to each class, and the class with the highest probability is chosen.

3. Python Implementation of Multinomial Logistic Regression

Step 1: Install Required Libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.datasets import load_iris

Step 2: Load the Iris Dataset

We will use the Iris dataset, which contains 3 classes of flowers:

• Setosa (Class 0)
• Versicolor (Class 1)
• Virginica (Class 2)

# Load dataset
iris = load_iris()
X = iris.data  # Features (sepal & petal measurements)
y = iris.target  # Target (0, 1, 2)

# Convert to DataFrame
df = pd.DataFrame(X, columns=iris.feature_names)
df['Target'] = y
print(df.head())

Step 3: Split Data into Training and Testing Sets

# Split data (80% training, 20% testing)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 4: Train the Multinomial Logistic Regression Model

# Create Multinomial Logistic Regression model
model = LogisticRegression(multi_class='multinomial', solver='lbfgs', max_iter=500)

# Train the model
model.fit(X_train, y_train)

Step 5: Make Predictions

# Predict on test data
y_pred = model.predict(X_test)

Step 6: Evaluate Model Performance

# Accuracy Score
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy)

# Confusion Matrix
conf_matrix = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:\n", conf_matrix)

# Classification Report
print("Classification Report:\n", classification_report(y_test, y_pred))

Step 7: Visualizing the Decision Boundaries

Since the dataset has 4 features, we use only 2 features (sepal length and sepal width) for visualization.

from mlxtend.plotting import plot_decision_regions

# Select two features for visualization
X_vis = X_train[:, :2]
y_vis = y_train

# Train model with selected features
model_vis = LogisticRegression(multi_class='multinomial', solver='lbfgs')
model_vis.fit(X_vis, y_vis)

# Plot decision boundaries
plot_decision_regions(X_vis, y_vis, clf=model_vis, legend=2)
plt.xlabel("Sepal Length (cm)")
plt.ylabel("Sepal Width (cm)")
plt.title("Multinomial Logistic Regression - Decision Boundary")
plt.show()

4. Understanding the Output

🔹 Accuracy Score → Percentage of correctly classified instances.
🔹 Confusion Matrix → Shows how well the model classified each class.
🔹 Classification Report → Precision, Recall, and F1-score for each class.
🔹 Decision Boundaries → Show how the model separates different classes.

Summary

✔ Multinomial Logistic Regression is for multi-class classification problems.
✔ Uses the Softmax function to calculate probabilities for each class.
✔ sklearn makes implementation easy in Python.
✔ Confusion Matrix & Accuracy Score help evaluate model performance.