Data Engineering

The work that happens between raw data and a trainable dataset. These pages are vendor-neutral — the techniques are statistics and scikit-learn, and they apply regardless of where the model eventually runs.

Start with Understanding data preparation, which covers the whole ground: missing data mechanisms and the imputation each one calls for, text cleaning, outlier detection, and imbalanced datasets.

Feature engineering is the overview of selection, extraction and transformation, and links out to the pages that cover each in depth.

Two of those pages are here: feature transformation and scaling for reshaping distributions and putting features on comparable scales, and data binning for turning continuous variables into discrete categories.

Encoding categorical variables and reducing dimensionality are covered under exploratory data analysis.