Data Science Feature Engineering

Feature Engineering is the process of using domain knowledge and data transformation techniques to create, modify, or select features (variables) that help machine learning models perform better. It is a key step in the data preparation process and can significantly impact model accuracy.

1. Why is Feature Engineering Important?

  • Improves Model Performance: Better features lead to more accurate and robust models.
  • Reveals Hidden Patterns: Transformed or derived features can uncover relationships not obvious in the raw data.
  • Reduces Overfitting: Proper feature selection and transformation can help prevent models from learning noise.
  • Simplifies the Model: By selecting or combining features, you can reduce model complexity while retaining important information.

2. Common Feature Engineering Techniques

2.1. Feature Creation (Derivation)

Creating new features from existing data can help capture underlying patterns.
Example:

  • Date Features: From a timestamp, derive day, month, or hour features. 
    import pandas as pd

# Sample data with a date column
data = pd.DataFrame({
    'timestamp': ['2025-01-01 08:00:00', '2025-01-02 12:30:00', '2025-01-03 15:45:00']
})

# Convert to datetime and extract features
data['timestamp'] = pd.to_datetime(data['timestamp'])
data['day'] = data['timestamp'].dt.day
data['month'] = data['timestamp'].dt.month
data['hour'] = data['timestamp'].dt.hour

print(data)

    Try It Now

2.2. Feature Transformation

Transforming features can improve their usefulness in models.

a) Scaling

  • Standardization: Rescale features so that they have a mean of 0 and a standard deviation of 1. 
    from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
data['scaled_feature'] = scaler.fit_transform(data[['day']])

    Try It Now

  • Normalization: Rescale features to a range of 0 to 1. 
    from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
data['normalized_feature'] = scaler.fit_transform(data[['hour']])

    Try It Now

b) Log Transformation

Useful when features have a skewed distribution.

import numpy as np

data['log_feature'] = np.log1p(data['day'])  # np.log1p handles zero values

Try It Now

2.3. Encoding Categorical Variables

Convert categorical variables into a numerical format that can be used by machine learning models.

a) Label Encoding

Assigns a unique number to each category.

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
data['category'] = ['A', 'B', 'A']  # Example categorical data
data['category_encoded'] = le.fit_transform(data['category'])

Try It Now

b) One-Hot Encoding

Creates binary columns for each category.

data = pd.get_dummies(data, columns=['category'])

Try It Now

2.4. Handling Date and Time Features

Extract meaningful features such as day of week, month, year, and whether the day is a weekend.

# Assuming 'timestamp' column is already converted to datetime
data['day_of_week'] = data['timestamp'].dt.dayofweek  # Monday=0, Sunday=6
data['is_weekend'] = data['day_of_week'].apply(lambda x: 1 if x >= 5 else 0)

Try It Now

2.5. Binning or Discretization

Converting continuous variables into discrete bins can simplify the model.

data['age'] = [23, 45, 31, 60, 18]
data['age_group'] = pd.cut(data['age'], bins=[0, 18, 35, 60, 100], labels=['Child', 'Young Adult', 'Adult', 'Senior'])
print(data[['age', 'age_group']])

Try It Now

2.6. Feature Interaction

Creating interaction features can capture the combined effect of two or more features.

# For example, creating an interaction term between 'age' and 'salary'
data['age_salary_interaction'] = data['age'] * 1000  # Assuming a hypothetical transformation

Try It Now

3. Best Practices for Feature Engineering

  • Understand Your Data: Explore and visualize your data before creating new features.
  • Domain Knowledge: Use insights from the domain to guide the creation of meaningful features.
  • Avoid Data Leakage: Ensure that engineered features do not include information from outside the training data (e.g., future information).
  • Iterate and Experiment: Try different feature transformations and evaluate their impact on your model performance.
  • Automate When Possible: Use pipelines to streamline the feature engineering process and ensure consistency.

4. Putting It All Together: A Simple Pipeline

You can combine several feature engineering steps into a pipeline for automation and reproducibility:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer

# Assume we have numerical and categorical features
num_features = ['day', 'hour']
cat_features = ['category']

num_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='mean')),
    ('scaler', StandardScaler())
])

cat_pipeline = Pipeline([
    ('encoder', OneHotEncoder(handle_unknown='ignore'))
])

preprocessor = ColumnTransformer([
    ('num', num_pipeline, num_features),
    ('cat', cat_pipeline, cat_features)
])

# Apply the preprocessing pipeline
processed_features = preprocessor.fit_transform(data)
print(processed_features)

Try It Now

 

Feature Engineering is a powerful technique in Data Science that transforms raw data into informative features, leading to better model performance and more accurate insights.