Area under ROC curve

Description

This function calculates the area under ROC curve

Usage

cal_auc(X, Y)

Arguments

Details

This function calculates the area under ROC curve.

Value

A numeric value of AUC will be returned.

References

https://www.r-bloggers.com/calculating-auc-the-area-under-a-roc-curve/

See Also

cal_confus()

Examples

data(test_data)
true_vec <- test_data[, 1]
pred_vec <- test_data[, 5]
confus_res <- cal_confus(true_vec, pred_vec)
AUC_res <- cal_auc(confus_res$TPR, confus_res$FPR)

Calculate confusion matrices

Description

This function calculates the confusion matrices across different cutoff points.

Usage

cal_confus(true_vec, pred_vec, force_diag=TRUE)

Arguments

Details

This function calculates the TP, FP, FN, TN, TPR, FPR and PPV across different cutoff points of pred_vec. TPR and FPR are forced to across (0, 0) and (1, 1) if force_diag=TRUE.

Value

References

https://en.wikipedia.org/wiki/Confusion_matrix

Examples

data(test_data)
true_vec <- test_data[, 1]
pred_vec <- test_data[, 5]
confus_res <- cal_confus(true_vec, pred_vec)

Multi-class classification PR

Description

This function calculates the Precision, Recall and AUC of multi-class classifications.

Usage

multi_pr(data, force_diag=TRUE)

Arguments

Details

A data frame is required for this function as input. This data frame should contains true label (0 - Negative, 1 - Positive) columns named as XX_true (e.g. S1_true, S2_true and S3_true) and predictive scores (continuous) columns named as XX_pred_YY (e.g. S1_pred_SVM, S2_pred_RF), thus this function allows calcluating ROC on mulitiple classifiers.

Predictive scores could be probabilities among [0, 1] and other continuous values. For each classifier, the number of columns should be equal to the number of groups of true labels. The order of columns won't affect results.

Recall, Precision, AUC for each group and each method will be calculated. Macro/Micro-average AUC for all groups and each method will be calculated.

Micro-average ROC/AUC was calculated by stacking all groups together, thus converting the multi-class classification into binary classification. Macro-average ROC/AUC was calculated by averaging all groups results (one vs rest) and linear interpolation was used between points of ROC.

AUC will be calculated using function cal_auc().

Value

Examples

data(test_data)
pr_test <- multi_pr(test_data)
pr_test$AUC 

Multi-class classification ROC

Description

This function calculates the Specificity, Sensitivity and AUC of multi-class classifications.

Usage

multi_roc(data, force_diag=TRUE)

Arguments

Details

A data frame is required for this function as input. This data frame should contains true label (0 - Negative, 1 - Positive) columns named as XX_true (e.g. S1_true, S2_true and S3_true) and predictive scores (continuous) columns named as XX_pred_YY (e.g. S1_pred_SVM, S2_pred_RF), thus this function allows calcluating ROC on mulitiple classifiers.

Predictive scores could be probabilities among [0, 1] and other continuous values. For each classifier, the number of columns should be equal to the number of groups of true labels. The order of columns won't affect results.

Specificity, Sensitivity, AUC for each group and each method will be calculated. Macro/Micro-average AUC for all groups and each method will be calculated.

Micro-average ROC/AUC was calculated by stacking all groups together, thus converting the multi-class classification into binary classification. Macro-average ROC/AUC was calculated by averaging all groups results (one vs rest) and linear interpolation was used between points of ROC.

AUC will be calculated using function cal_auc().

Value

References

http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html

Examples

data(test_data)
roc_test <- multi_roc(test_data)
roc_test$AUC 

Generate PR plotting data

Description

This function generates plotting PR data for following data visualization.

Usage

plot_pr_data(pr_res)

Arguments

Value

Examples

data(test_data)
pr_res <- multi_pr(test_data)
pr_res_df <- plot_pr_data(pr_res)

Generate ROC plotting data

Description

This function generates plotting ROC data for following data visualization.

Usage

plot_roc_data(roc_res)

Arguments

Value

Examples

data(test_data)
roc_res <- multi_roc(test_data)
roc_res_df <- plot_roc_data(roc_res)

Output of PR bootstrap confidence intervals

Description

This function uses bootstrap to generate five types of equi-tailed two-sided confidence intervals of PR-AUC with different required percentages and output a dataframe with AUCs, lower CIs, and higher CIs of all methods and groups.

Usage

pr_auc_with_ci(data, conf= 0.95, type='bca', R = 100)

Arguments

Details

A data frame is required for this function as input. This data frame should contains true label (0 - Negative, 1 - Positive) columns named as XX_true (e.g. S1_true, S2_true and S3_true) and predictive scores (continuous) columns named as XX_pred_YY (e.g. S1_pred_SVM, S2_pred_RF). Predictive scores could be probabilities among [0, 1] and other continuous values. For each classifier, the number of columns should be equal to the number of groups of true labels. The order of columns won't affect results.

Value

Examples

## Not run: data(test_data)
pr_auc_with_ci_res <- pr_auc_with_ci(test_data, conf= 0.95, type='bca', R = 100)
## End(Not run)

PR bootstrap confidence intervals

Description

This function uses bootstrap to generate five types of equi-tailed two-sided confidence intervals of PR-AUC with different required percentages.

Usage

pr_ci(data, conf= 0.95, type='basic', R = 100, index = 4)

Arguments

Details

A data frame is required for this function as input. This data frame should contains true label (0 - Negative, 1 - Positive) columns named as XX_true (e.g. S1_true, S2_true and S3_true) and predictive scores (continuous) columns named as XX_pred_YY (e.g. S1_pred_SVM, S2_pred_RF). Predictive scores could be probabilities among [0, 1] and other continuous values. For each classifier, the number of columns should be equal to the number of groups of true labels. The order of columns won't affect results.

Value

Examples

## Not run: data(test_data)
pr_ci_res <- pr_ci(test_data, conf= 0.95, type='basic', R = 1000, index = 4)
## End(Not run)

Output of ROC bootstrap confidence intervals

Description

This function uses bootstrap to generate five types of equi-tailed two-sided confidence intervals of ROC-AUC with different required percentages and output a dataframe with AUCs, lower CIs, and higher CIs of all methods and groups.

Usage

roc_auc_with_ci(data, conf= 0.95, type='bca', R = 100)

Arguments

Details

A data frame is required for this function as input. This data frame should contains true label (0 - Negative, 1 - Positive) columns named as XX_true (e.g. S1_true, S2_true and S3_true) and predictive scores (continuous) columns named as XX_pred_YY (e.g. S1_pred_SVM, S2_pred_RF). Predictive scores could be probabilities among [0, 1] and other continuous values. For each classifier, the number of columns should be equal to the number of groups of true labels. The order of columns won't affect results.

Value

Examples

## Not run: data(test_data)
roc_auc_with_ci_res <- roc_auc_with_ci(test_data, conf= 0.95, type='bca', R = 100)
## End(Not run)

ROC bootstrap confidence intervals

Description

This function uses bootstrap to generate five types of equi-tailed two-sided confidence intervals of ROC-AUC with different required percentages.

Usage

roc_ci(data, conf= 0.95, type='basic', R = 100, index = 4)

Arguments

Details

A data frame is required for this function as input. This data frame should contains true label (0 - Negative, 1 - Positive) columns named as XX_true (e.g. S1_true, S2_true and S3_true) and predictive scores (continuous) columns named as XX_pred_YY (e.g. S1_pred_SVM, S2_pred_RF). Predictive scores could be probabilities among [0, 1] and other continuous values. For each classifier, the number of columns should be equal to the number of groups of true labels. The order of columns won't affect results.

Value

Examples

## Not run: data(test_data)
roc_ci_res <- roc_ci(test_data, conf= 0.95, type='basic', R = 1000, index = 4)
## End(Not run)

Example dataset

Description

This example dataset contains two classifiers (m1, m2), and three groups (G1, G2, G3).

Usage

data("test_data")

Format

A data frame with 85 observations on the following 9 variables.

G1_true

true labels of G1 (0 - Negative, 1 - Positive)

G2_true

true labels of G2 (0 - Negative, 1 - Positive)

G3_true

true labels of G3 (0 - Negative, 1 - Positive)

G1_pred_m1

predictive scores of G1 in the classifier m1

G2_pred_m1

predictive scores of G2 in the classifier m1

G3_pred_m1

predictive scores of G3 in the classifier m1

G1_pred_m2

predictive scores of G1 in the classifier m2

G2_pred_m2

predictive scores of G2 in the classifier m2

G3_pred_m2

predictive scores of G3 in the classifier m2

Examples

data(test_data)