How should detecting and removing outliers be done in a dataset containing data of multiple dimensions?.TL DR: I have two questions regarding data preprocessing. It still remains as a question whether I made a proper approach or not. Compared to the performance of the model trained using the original dataset, the model trained using the augmented dataset did not show significant improvement in performance. The dataset mainly consists of augmented data training the model centering on artificial data may lead to unwanted results. The original dataset does not contain much quantity of data for training.I did realize this, but decided to continue as removing the outliers will help in retrieving a more accurate regression function illustrating the relationship between the target data and each feature data. Despite the randomly generated target data values are within the IQR range, other feature data values derived from the regression may exist outside the IQR range thus, making the outlier elimination process to be meaningless.
There are two main fallacies in the data augmentation approach I made. And using the augmented data set, I trained the model and made predictions. Then, I randomly generated target data values that lie within the IQR range used the previously discovered regression relationship to get other feature data values. Thus, I created my own approach.Īfter removing the outliers from the dataset, I plotted a polynomial regression function in order to find the relationship between the target data (data to be predicted) and each individual feature data (data to be used in training the model) - how a certain target data value relates to a certain feature data.
While a dedicated library exists for augmenting image datasets by randomly modifying its structure, it's challenging to find an approach for augmentation of numerical data. Now I would like to ask about data augmentation. As seven different columns are grouped in a row, I thought that looping through each column and removing the outliers would result in a dataframe containing the data within the IQR range, even though its overall quantity may have been reduced. I looped through each individual column and removed the row that contains the data that exist outside the IQR range. When illustrated on a dataframe, there are seven different columns each row acting as a metadata explaining the properties of a single data. Here's my approach: the dataset consisted seven different dimensions of data. But how should the process of detecting and removing outliers be done if the dataset is composed of multiple dimensions of data? Removing the outliers of a single-dimensional data can be easily done by removing the points that are outside of the IQR range.
0 kommentar(er)
