Estimators vs transformers: the fit, transform and predict contracts

1.Estimator is the umbrella term: everything learns with fit

scikit-learn is the default single-node ML library on Databricks. It is included in Databricks Runtime ML, and the exam expects you to know how its API is organised. Everything in that API rests on one word: estimator. Candidates often treat "estimator" as another word for "model that predicts". That is too narrow. In scikit-learn, an estimator is any object that learns from data through a fit method. What it does after fitting decides which kind of estimator it is.

The answer is yes. A scaler has to learn the mean and standard deviation of each column before it can rescale anything, and learning from data is what makes something an estimator. There are two families you need to tell apart:

- Transformers learn a mapping from data in fit, then apply it with transform and return a new feature matrix. Examples are scalers, imputers and PCA. - Predictors learn a relationship between X and a target y in fit, then produce outputs with predict. Examples are classifiers and regressors such as LogisticRegression or SVC. Classifiers usually also offer predict_proba for class probabilities.

The scikit-learn documentation uses this vocabulary throughout. When it describes composite models, it says transformers are combined "with other transformers or with predictors", which treats both as kinds of estimator. Many estimators also have a score method that gives a default evaluation criterion. Scoring belongs to the evaluation lessons, so we only note here that it is part of the estimator's interface too.

Sources123

2.Transformers: fit learns the statistics, transform applies them

Raw data almost never goes straight into a model. Databricks' own feature engineering guidance notes that raw data usually needs preprocessing and transformation before a model can be built from it. In scikit-learn, that preprocessing is done by transformers, and the idea that matters is that a transformer has state.

Calling fit on a StandardScaler computes each column's mean and standard deviation and stores them on the object. Fitted attributes end in an underscore, such as mean_ and scale_. Calling transform then applies those stored numbers to whatever X you pass in. The two calls are separate on purpose: you fit once, on training data, and then transform as many datasets as you need.

>>> scaler = preprocessing.StandardScaler().fit(X_train)
>>> scaler
StandardScaler()

The training set's. That is the point of keeping fit and transform apart: the test data is scaled with exactly the shifts and scales learned during fit. If you called fit again on the test set, you would get a different transformation, and your evaluation would no longer reflect how the model behaves on unseen data. Imputers follow the same contract. SimpleImputer learns a fill value per column, such as the mean, in fit, and fills gaps with those learned values in transform.

>>> imp = SimpleImputer(missing_values=np.nan, strategy='mean')
>>> imp.fit([[1, 2], [np.nan, 3], [7, 6]])
SimpleImputer()
>>> X = [[np.nan, 2], [6, np.nan], [7, 6]]
>>> print(imp. ? (X))

Sources45

3.Predictors, and why a Pipeline cares which is which

A predictor also learns in fit, but it learns a relationship to a target y, and its output is a prediction, not a new feature matrix. Regressors return values from predict. Classifiers return labels from predict and usually probabilities from predict_proba.

The difference between transformers and predictors is not just naming. sklearn.pipeline.Pipeline enforces it. A pipeline is a chain of estimators, and it only works if every step except the last produces data the next step can consume. So every step except the last must be a transformer. The last step can be a predictor, a transformer, or another kind of estimator.

Calling fit on a pipeline is the same as calling fit on each step in turn. Each transformer is fitted, its output is transformed and passed on, and finally the predictor is fitted on fully transformed data. At prediction time, the fitted transformers only transform. The example below chains a transformer and a predictor. One fit call learns the scaling statistics and the classifier coefficients from the training data alone.

>>> pipe = make_pipeline(StandardScaler(), LogisticRegression())
>>> pipe.fit(X_train, y_train)  # apply scaling on training data
Pipeline(steps=[('standardscaler', StandardScaler()),
                ('logisticregression', LogisticRegression())])

Sources32

4.The same words in Spark MLlib, with a twist

Databricks supports classic ML with both scikit-learn and Apache Spark MLlib, and the exam can use the terms "estimator" and "transformer" for either. The ideas carry over, but in the pyspark.ml Pipelines API the relationship between the two terms is arranged differently. Learn the difference explicitly:

- A Spark Transformer has a transform() method that takes a DataFrame and returns a new DataFrame, usually with extra columns appended. - A Spark Estimator has a fit() method that takes a DataFrame and returns a Model, and that Model is itself a Transformer. - Prediction in Spark is just another transform(). Calling transform on a fitted classification model appends a prediction column. The Pipelines API has no separate predict step.

The twist is this. In scikit-learn, a fitted StandardScaler is the same object you created, now carrying mean_ and scale_. In Spark, StandardScaler is an Estimator, and its fit() returns a separate model object that does the transforming. Likewise, a Spark Pipeline is an Estimator, and fitting it returns a PipelineModel, which is a Transformer. In both libraries, a learned step is fitted once on training data and then applied unchanged to new data. Databricks' MLlib examples include building a custom transformer, which is the extension point when the built-in feature stages are not enough.

Sources674

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1."Estimator" means a model that makes predictions, so scalers and imputers are not estimators.Why is that wrong?

   Estimator is the umbrella term for anything that learns with fit. Transformers are estimators that expose transform, and predictors are estimators that expose predict.

   Covered in Estimator is the umbrella term: everything learns with fit

2. 2.To scale test data correctly, call fit_transform on the test set so it is scaled by its own statistics.Why is that wrong?

   A transformer is fitted on training data only. The same fitted instance then calls transform on test or new data, so both get the identical transformation.

   Covered in Transformers: fit learns the statistics, transform applies them

3. 3.Every Pipeline has a predict method.Why is that wrong?

   A Pipeline has exactly the methods of its last step. If the last step is a transformer such as PCA, the pipeline behaves as a transformer and has no predict.

   Covered in Predictors, and why a Pipeline cares which is which