Feature engineering is the crucial process of transforming raw data into features that expose informative relationships and patterns to machine learning models. Though often time-consuming, proper feature engineering can dramatically improve model accuracy and performance. In this comprehensive guide, we‘ll explore common techniques, challenges, benefits, and automation approaches for effective feature engineering.
What is Feature Engineering and Why it Matters
Feature engineering refers to the set of techniques used to convert raw data into formats that can be effectively processed by machine learning algorithms. The engineered features are designed to amplify signal and bring out insightful relationships in the data.
This process is a key part of applied machine learning. In a survey by Kaggle, data scientists and ML engineers reported spending over 40% of their project time on collecting, cleaning, and feature engineering data [1]. Of this, feature engineering constituted a significant portion.
Another study found that proper feature engineering could improve model AUC by over 10% across multiple datasets [2]. The choice and quality of features provided to a model can make or break its performance.
| Reference | [1] Kaggle, "State of Data Science and Machine Learning", 2017 |
| Reference | [2] Kanter et al. "Deep Feature Synthesis: Towards Automating Data Science Endeavors", IEEE DSAA, 2015 |
Some key reasons why feature engineering is invaluable:
-
Enables simpler models: Well-engineered features allow linear/shallower models to capture complex relationships that would require deep neural networks on raw data.
-
Avoids overfitting: Irrelevant features contribute to overfitting. Feature engineering retains meaningful signals and discards noise.
-
Speeds up training: Additional noisy features slow down training. Reducing dimensionality speeds convergence.
-
Improves accuracy: Feature transformations can expose latent relationships that are non-obvious in the raw data.
-
Enables new kinds of predictions: Aggregations, ratios, and interactions between features enables predicting new targets.
The next sections explore common techniques for engineering impactful features.
Encoding Categorical Variables
Most machine learning models require numerical feature inputs. Categorical variables like gender, genres, departments, etc take on categorical values. To input them into models, we must convert these categories into numerical representations.
One-Hot Encoding
One-hot encoding (or dummy encoding) converts each unique category value into a new binary variable that indicates the presence (or absence) of that category.
For example, encoding a "Department" variable with values {Engineering, Sales, Marketing} would generate 3 new binary variables:
- Is_Engineering (1 if Engineering, else 0)
- Is_Sales (1 if Sales, else 0)
- Is_Marketing (1 if Marketing, else 0)
This allows the model to learn independent weights for each department.
Ordinal Encoding
Ordinal encoding assigns each unique category value an integer code like:
- Engineering -> 0
- Sales -> 1
- Marketing -> 2
This assumes an ordinal relationship between categories (Sales > Engineering), which may not apply. Use carefully.
Embeddings
Embeddings map categories to dense vector representations in a lower-dimensional space. Proximity between vectors captures semantic similarity between categories.
Embeddings can group similar departments near each other and dissimilar ones further apart. This extra information helps the model learn richer representations.
Transforming Numerical Features
For numerical features, common transformations include:
Discretization
Continuous numerical variables are binned into discrete ranges or categories:
- Age: 0-17, 18-35, 36-50, 51-65, 65+
- Income: Low, Medium, High
This exposes non-linear relationships.
Scaling
Feature values are shifted and rescaled to a standard range (eg. 0 to 1). This accounts for differences in value ranges and brings all features to comparable scales.
For example, normalizing units of measurement like height (in feet vs meters).
Log / Power Transforms
Logged or powered transformations help normalize skewed distributions and reduce impact of outliers.
For example, log-transforming home prices or square-root total sales.
aggregation
Statistics like mean, standard deviation, min/max, etc. summarize a group of values into a single descriptive metric.
For example, deriving average transaction amount of customers over past 30 days.
Differencing
Instead of absolute values, encoding change between current and past values (delta features) removes trends over time.
For example, using 7-day increase in sales rather than absolute weekly sales.
Constructing Informative New Features
Domain expertise guides combining raw inputs into more informative derived features.
Ratios
Dividing two raw inputs creates relative scales and ratios. For example:
- Signal-to-noise ratio
- Debt-to-equity ratio
- Return on investment
Captures significant relationships obscured in individual variables.
Interactions
Multiplying raw inputs models interactions between features. For example:
- Performance = Skill * Motivation
- Revenue = Traffic * Conversion Rate
Expresses compounded effects of factors.
Aggregations
Apply aggregation functions like mean, min, max, etc. on groups of records.
For example, average spend per customer over lifetime.
Summarizes a collection of values into descriptive statistics.
Lagged Features
Prior time period values of variables provide useful context. For example:
- 7-day lagged sales
- 60-day lagged returns
Helps identify trends not visible in current values alone.
Feature Selection
Having too many noisy, irrelevant features slows down training and leads to overfitting. Feature selection identifies and retains only useful subsets of features.
Filter Methods
Filter methods use statistical metrics like information gain, correlation coefficients etc. to rank and shortlist impactful features.
They evaluate features independently based on relevance to the target variable. Fast to compute but can miss relationships between features.
Wrapper Methods
Wrapper methods search for optimal feature subsets by using a model‘s performance as the evaluation metric.
For example, evaluating different subsets based on validation accuracy of a logistic regression model. Captures feature interactions better but more computationally intensive.
Embedded Methods
Some model construction techniques perform feature selection automatically as part of the training process.
For example, LASSO regression penalizes model complexity helping eliminate weak features. Decision tree algorithms also incorporate feature selection.
Dimensionality Reduction
High-dimensional datasets with many features can exhibit poor model performance and overfitting issues. Dimensionality reduction condenses variables into lower dimensions while preserving most information.
Principal Component Analysis (PCA)
PCA transforms original features into orthogonal principal components that capture decreasing proportions of variance.
Selecting just the top few components retains most information while removing redundant and noisy inputs.
Autoencoders
Autoencoders are neural networks trained to encode inputs into lower-dimensional representations and then reconstruct the original inputs.
The compact encoded representations retain only the most useful facets of the data.
Challenges in Feature Engineering
While critical, proper feature engineering can be difficult. Common challenges include:
- Dealing with noise, outliers and missing values in raw data
- Leakage – allowing data about the target leak into the feature set
- Under-transforming variables not exposed to enough transformations
- Oversaturating with unnecessary constructed features
- Imbalanced processing of different feature types
- Difficulty selecting optimal subsets from large feature sets
- Lack of reproducibility in manual feature engineering
These underline the need for rigorous evaluation of engineered features before selection.
Walkthrough of Feature Engineering Process
Let‘s walk through the full process on a supervised learning task:
Problem: Predict customer repeat purchase likelihood from e-commerce transaction history
Data: Customer ID, email domain, items purchased, order timestamps, cost, category, brand, returns etc.
Target: Repeated again in 60 days (1/0)
We engineer features in stages:
-
Clean missing values and outliers, address class imbalance
-
Encode categoricals like item category, brand using one-hot encoding
-
Transform numerical features like cost using scaling and log transforms
-
Derive new aggregations like average spend, time between purchases, purchase frequency
-
Interact brand x category combinations to capture relationships
-
Reduce dimensions with PCA and autoencoders on cleaned numerical attributes
-
Select via RFE and tree algorithms to retain top features
-
Evaluate impact on model metrics and refine engineered features
This exercise utilized both domain knowledge and technical skills to systematically craft impactful features. Iterating based on evaluation feedback leads to the best set of engineered inputs.
Automating Feature Engineering
Manually engineering useful features requires significant time and effort. As a result, tools are emerging to automate parts of the process using data and ML:
- Featuretools auto-generates feature sets by analyzing relationships in the dataset
- AutoML solutions automate feature engineering, selection, and model building
- Meta-learning recommends transformations based on past task performance
However, automated feature engineering still requires human guidance. Domain knowledge is crucial for constructing relevant derived features. Iteratively evaluating feature impact and providing feedback helps auto-tools focus on useful transformations.
The future of feature engineering is combining automation with human ingenuity and oversight.
Key Takeaways
The main lessons are:
- Proper feature engineering substantially improves ML model performance
- Transforming raw data into informative representations is crucial
- Domain expertise guides creative application of techniques like embeddings, new feature construction etc.
- Feature selection retains only useful signals and discards redundancies
- Dimensionality reduction removes noise while preserving information
- Avoiding data leakage, under- and over-engineering takes practice
- Automation complements manual feature engineering but doesn‘t replace it
Feature engineering is a complex iterative process but worth the effort. Learning both its technical and intuitive aspects is key for applying ML successfully.