Question

I’m new to ML with scikit on jupyter notebook, so i was wondering after you train a classifier from your original dataset that was split into train and test dataset, how do I test a completely new dataset to this previously trained classifier?

Answer 1

If you want to convert an existing Python function into a transformer to assist in data cleaning or processing. You can implement a transformer from an arbitrary function with FunctionTransformer. For example, to build a transformer that applies a log transformation in a pipeline, do:

>>> import numpy as np
>>> from sklearn.preprocessing import FunctionTransformer
>>> transformer = FunctionTransformer(np.log1p, validate=True)
>>> X = np.array([[0, 1], [2, 3]])
>>> transformer.transform(X)
array([[0. , 0.69314718],
[1.09861229, 1.38629436]])

You can ensure that func and inverse_func are the inverse of each other by setting check_inverse=True and calling fit before transform. Please note that a warning is raised and can be turned into an error with a filterwarnings:

>>> import warnings
>>> warnings.filterwarnings("error", message=".*check_inverse*.",
... category=UserWarning, append=False)

Now you can again load new dataset :

Let’s load the dataset using pandas.

>>> data=pd.read_csv('newdataset.csv')
>>> data.head()