This post will explore using R’s MLmetrics to evaluate machine learning models. MLmetrics provides several functions to calculate common metrics for ML models, including AUC, precision, recall, accuracy, etc.
Building an example model
Firstly, we need to build a model to use as an example. For this post, we’ll be using a dataset on pulsar stars from Kaggle. Let’s save the file as “pulsar_stars.csv”. Each record in the file represents a pulsar star candidate. The goal will be to predict if a record is a pulsar star based upon the attributes available.
To get started, let’s load the packages we’ll need and read in our dataset.
library(MLmetrics)
library(dplyr)
stars = read.csv(“pulsar_stars.csv”)
Next, let’s split our data into train vs. test. We’ll do a standard 70/30 split here.
set.seed(0)
train_indexes = sample(1:nrow(stars), .7 * nrow(stars))
train_set <- stars[train_indexes,]
test_set <- stars[-train_indexes,]
Now, let’s build a simple logistic regression model.
train_set <- data.frame(train_set %>% select(target_class), train_set %>% select(-target_class))
# build model
model <- glm(formula(train_set), train_set, family = "binomial")
# get predictions on train and test datasets
train_pred <- predict(model, train_set, type = "response")
test_pred <- predict(model, test_set, type = "response")
AUC / precision / recall / accuracy
Let’s calculate a few metrics. One of the most common metrics for classification is calculating AUC, which can be done using MLMetrics’ AUC function. Intuitively, AUC is a score between 0 and 1 that measures how well a model rank-orders predictions. See here for a more detailed explanation.
# get AUC on test and train set
AUC(test_pred, test_set$target_class) # 0.974172
AUC(train_pred, train_set$target_class) # 0.9773794
As a refresher, here’s a quick overview of precision, recall, and accuracy:
- Precision: The true positive rate. If the model predicts there are 10 pulsar stars, and 8 of those 10 actually are pulsars, then the precision would be 8 / 10, or 80%.
- Recall:The proportion of the positive labels that are captured with the model. For example, suppose there are 10 pulsar stars in the data and that the model predicts 7 of those to be pulsar stars. That would mean the recall is 7 / 10, or 70%.
- Accuracy:Generally the most intuitive of the metrics here. Accuracy is simply the number of correct predictions divided by the total number of predictions.
Visit TheAutomatic.net to read the full article:
http://theautomatic.net/2020/01/29/evaluate-your-r-model-with-mlmetrics/
Disclosure: Interactive Brokers
Information posted on IBKR Traders’ Insight that is provided by third-parties and not by Interactive Brokers does NOT constitute a recommendation by Interactive Brokers that you should contract for the services of that third party. Third-party participants who contribute to IBKR Traders’ Insight are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.
This material is from TheAutomatic.net and is being posted with permission from TheAutomatic.net. The views expressed in this material are solely those of the author and/or TheAutomatic.net and IBKR is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.
In accordance with EU regulation: The statements in this document shall not be considered as an objective or independent explanation of the matters. Please note that this document (a) has not been prepared in accordance with legal requirements designed to promote the independence of investment research, and (b) is not subject to any prohibition on dealing ahead of the dissemination or publication of investment research.
Any trading symbols displayed are for illustrative purposes only and are not intended to portray recommendations.
