Facies Prediction from Well Logs - Neural Network Model Hyper-Parameters Optimisation

In this project, we will implement standard and comprehensive steps to select the best model and hyper prameters to predict rock facies form a dataset. First, we will prepare data for modeling, fit the models and cross validate, predict facies labels and evaluate prediction accuracy by several model evaluation metrics. Finally, we will examine model performance on blind-well data. These are the models that we will use:
1 - Logistic Regression Classifier
2 - K Neighbors Classifier
3 - Decision Tree Classifier
4 - Random Forest Classifier
5 - Support Vector Classifier
6 - Gaussian Naive Bayes Classifier
7 - Gradient Boosting Classifier
8 - Extra Tree Classifier

The dataset for this study comes from Hugoton and Panoma Fields in North America. It consists of log data (the measurement of physical properties of rocks) of nine wells. We will use these log data to train supervised classifiers in order to predict discrete facies groups. For more detail, you may take a look here. The seven features are:

GR: this wireline logging tools measure gamma emission
ILD_log10: this is resistivity measurement
PE: photoelectric effect log
DeltaPHI: Phi is a porosity index in petrophysics.
PNHIND: Average of neutron and density log.
NM_M:nonmarine-marine indicator
RELPOS: relative position

The nine discrete facies (classes of rocks) are:
(SS) Nonmarine sandstone
(CSiS) Nonmarine coarse siltstone
(FSiS) Nonmarine fine siltstone
(SiSH) Marine siltstone and shale
(MS) Mudstone (limestone)
(WS) Wackestone (limestone)
(D) Dolomite
(PS) Packstone-grainstone (limestone)
(BS) Phylloid-algal bafflestone (limestone)

The project content:

1- Data Exploratory Analysis¶

1-1 Data visualization

1-1-1 log-plot
1-1-2 Bar plot
1-1-3 Cross-plot

1-2 Feature Engineering

1-2-1 NaN imputation
1-2-2 Feature extraction
1-2-3 Oversampling

1-3 Feature Importance

1-3-1 Feature linear corrolation
1-3-2 Decision tree
1-3-3 Permutation feature importance

2- Build Model & Validate¶

2-1 Baseline Model
2-2 Hyper-parameters

2-2-1 Grid search

3- Model Evaluation-1¶

3-1 Model Metrics
3-2 Confusion matrix

4- Model Evaluation-2¶

4-1 Learning curves
4-2 ROC plot
4-3 Blind well prediction and evaluation

1 Data Exploratory Analysis¶

After data reading into python using Pandas, we can visualize it to understand data better. Before plotting, we need to define a color map(this step deserves to be in the Feature engineering part but we need here to plot color for facies classes) and devote color code for each facies.

Note1: codes embedded in this manuscript are presented to understand the work procedure. If you want to exercise by yourself, I highly recommend using the jupyter notebook file.

Note2: shuffling data can cause differences between your runs and what appears here.

1-1 Data visualization¶

1-1-1 log-plot¶

This is function to create a plot.

Call function to plot

1-1-2 Histogram¶

We can use the Counter function to evaluate each class contribution quantitatively. To see facies frequency distribution we can use a bar plot as:

This is imbalanced dataset. Dolomite has the lowest member participation. Comparing coarse siltstone, dolomite appears 8 times less than that.

1-1-3 Cross-plot¶

To visualize multiple pairwise bivariate distributions in a dataset, we may use the pairplot() function from the seaborn library. It shows the relationship for the combination of variables in the dataset in the matrix format with a univariate distribution plot in diagonal. It is clear that PE log has a non-linear relationship with average porosity. Other pairs do not show a clear pattern. The distribution pattern in diagonal shows that each label class (facies) with respect to each feature has acceptable separation although there is a strong overlap for various classes. The ideal pattern can be assumed as a clear separation of distribution plots in tall bell shape normal distribution graph.

Hereafter we will store dataset into new vriable after main operations (indented paragrpahs in introduction).¶

1-2 Feature Engineering¶

1-2-1 NaN imputation¶

Let's look at the data to see if Null values are present in dataset.

Here, PE has 917 null values.

There are several ways to deal with Null values in the dataset. The simplest approach is to drop the rows containing at least one null value. This can be logical with a bigger size dataset but in small data frames, single points are important. We can impute null values with mean or from adjacent data points in columns. Filling with mean value will not affect data variance and therefore will not have an impact on prediction accuracy, though can create data bias. Filling with the neighbor cells of column values can be appropriate if we have a geologically homogeneous medium like mass pure carbonate rocks.

Another approach, that I will implement here, to employe machine learning models to predict missing values. This is the best way of dealing with this dataset because we have just a single feature missing from the dataset, PE. On the other hand, filling with ML prediction is much better than the single mean value because we are able to see ML correlation and accuracy by dividing data to train and test sets.

We can use various ML models to predict PE log as countinious regression problem. Here, I will employ Multi-Layer Percepteron Neural Network from scikit-learn to predict target value. I am not going to deep for this approach and use simply to predict missing values.

Plot predecited PE¶

Predicted PE in well ALEXANDER D shows the normal range and variation. Prediction accuracy is 77%.

1-2-2 Feature extraction¶

Having limited set of features in this dataset can lead us to think about extracting some data from existing dataset.

First, we can convert formation categorical data into numeric data. Our background knowledge can help us to predict that some facies are possibly presnet more in a specififc formation rather than others.

We can use the LabelEncoder function:

We converted formation category data into numeric to use as a predictor and added 1 to start predictor from 1 instead of zero. To see if new feature extraction would assist prediction improvement, we should define a baseline model then compare it with the extracted feature model.

Pick up a well as blind:¶

machine learning algorithm will not see this data in the training process. We will use it at the end to see how the model works. Remember to select a well that includes all types of facies classes, otherwise in prediction, data dimension inconsistency will generate an error. Or simply add a lacking facies example to the well to avoid the problem.

To see if new feature extraction would assisst prediction improvment, we should define a baseline model then compare with extracted feature model.

Baseline Model Performance¶

For simplicity, we will use a logistic regression classifier as a baseline model and will examine model performance with a cross-validation concept. Data will be split into 10 subgroups and the process will be repeated 3 times.

For simplicity we will use logistic regression classifier as baseline model and will examine model performance with cross validation concept. Data will be splitted into 10 subgroups and process will be reapeted 3 times.

Here, we can explore whether feature extraction can improve model performance. There are many aprroaches while we will use some transforms for chaining the distribution of the input variables such as Quantile Transformer and KBins Discretizer. Then, will remove linear dependecies between the input variables using PCA and TruncatedSVD. To study more refer here.
Using feature union class, we will define list of transforms to perform results aggrigated together. This will create a dataset with lots of feature columns while we need to reduce dimentionality to faster and better performance. Finally, Recursive Feature Elimination, or RFE, technique can be used to select the most relevent features. We select 30 features.

Accuracy improvement shows that feature extraction can be useful approach when we are dealing with limited features in dataset.

1-2-3 Oversampeling¶

In imbalanced datasets, we can use resampling technique to add some more data point to increase members of minority groups. This can be helpful whenever minority label targets has special importance such as credit card fraud detection. In that example, fraud can happen with less than 0.1 percent of transaction while it is important to detect fraud.
In this work, we will add psudo observation for Dolomite class which has the lowest poulation

Synthetic Minority Oversampling Technique, SMOTE: the technique is used to select nearest neighbors in the feature space, separated examples by adding a line and producing new examples along the line. The method is not merely generating the duplicates from the outnumbered class, but applied K-nearest neighbours to generate synthetic data.

Now, dataset is balanced. Let's see how works comparing Baseline model:

Accuracy improved by 3 percent but in multi-class classification, accuracy is not the best evaluation metrics. We will cover others in next parts.

1-3 Feature Importance¶

Some machine learning algorithms offer importance score to help the user to select the most efficient features for prediction.

1-3-1 Feature linear corrolation¶

The concept is simple: features have higher correlation coefficient with target values are important for prediction. We can extract these coef's like:

1-3-2 Decision tree¶

This algorithm provides importance scores based on the reduction in the criterion used to split in each node such as entropy or Gini.

XGBoost is a library that provides an efficient and effective implementation of the stochastic gradient boosting algorithm. This algorithm can be used with scikit-learn via the XGBRegressor and XGBClassifier classes.

1-3-3 Permutation feature importance¶

Permutation feature importance is a model inspection technique that can be used for any fitted estimator when the data is tabular. This is especially useful for non-linear or opaque estimators. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled.

In all these feature importance plots we can see that predictor number 6 (PE log) has the most importance in label prediction. Based on the model that we select to evaluate the result, we may choose features based on their importance and neglect the rest to speed up the training process. This is very common if we are rich in feature quantity, though in our example dataset here, we will use all features as predictors are limited.

2 Build Model & Validate¶

2-1 Baseline Model¶

The philosophy of constructing a baseline model is simple: we need a basic and simple model to see how the adjustments on both data and model parameters can cause improvement in model performance. In fact, this is like a scale for comparison.

In this code script, we first defined our favorite model classifiers. Then, established baseline_model function. In this function, we employed the Pipeline function to implement step wised operation of data standard scaling(facilitate model running more efficient) and model object calling for cross-validation. I like Pipeline because it makes the codes more tidy and readable.

Define a function that uses pipeline to impelement data transformation and fit with model then cross validate

Run functions

Cross-validation#:~:text=Cross%2Dvalidation%2C%20sometimes%20called%20rotation,to%20an%20independent%20data%20set.) sometimes called rotation estimation or out-of-sample testing is any of the various similar model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set. Models usually are overfitting when the accuracy score on training data is much higher than testing data. One way to examine model performance is to keep randomly some part of the dataset hold-out. This can be a weakness for small datasets. Another way is to divide the dataset into splits and run it while each split has a different set of test folds like the picture below. In this approach, cross-validation, models can examine all data without overfitting. However, for this project, we will keep a single well as a hold-out to examine model performances after all optimizations.

picture from scikit-learn

We repeated 3 times each operation over a dataset that already divided into 10 equal parts(folds) in cross-validation. In fact, we intended to expose the model to all data with a different combination of train and test sets without overlap.

Here, we used an average of accuracy as metrics for comparison of various model performances(accuracy and other evaluation metrics will be elaborated in the next post). It is used for simplicity, while for multi-class classification problems, accuracy is the weakest model evaluation approach. We will cover model evaluation metrics for multi-class classification problems in the next posts.

The Extra tree and Random forest classifier showed the best accuracy score for the facies label prediction while the Gaussian Naive Bayes classifier performed poorly.

2-2 Hyper-parameters¶

2-2-1 Grid search¶

We created a baseline model in the previous section without adjucting hyper-parameters (parameters that are adjustable by user). Sometimes, careful selection of these parameters can improve model results noticeably. Grid search is designed to employ numeric/string range for specific hyper parameter without having to code loop functions.