Concept · intermediate
Evaluation metrics
Measure a classifier by the mistakes it makes, not just how often it's right.
Most pages on this site end with an accuracy: the share of predictions a model gets right. That’s a fine first number, but it treats every mistake alike and hides what the model actually does. Evaluation metrics split the mistakes apart so you can see them.
The dataset
data.csv holds 200 parts from a production line. defective is 1 if the
part really is defective and 0 if it’s good, and score is what a model
said, from 0 to 1, with higher meaning “more likely defective”. The data
is synthetic (see make_data.py), and 33 of the 200 parts are defective.
A score is not yet a decision. You choose a threshold and flag every part whose score reaches it.
Four counts
Every part lands in one of four boxes, the confusion matrix. At a threshold of 0.5:
| Flagged | Not flagged | |
|---|---|---|
| Defective | 15 true positives | 18 false negatives |
| Good | 6 false positives | 161 true negatives |
Every metric on this page is a ratio of those counts:
Precision asks: of the parts we flagged, how many were defective? Here, 15 of 21, or 0.71. Recall asks: of the defective parts, how many did we flag? Here, 15 of 33, or 0.45. F1 combines the two into one number, their harmonic mean, which is 0.56 here and stays low unless both are high.
Why accuracy misleads
At 0.5, the model is right about 176 of 200 parts, an accuracy of 0.88. That sounds good until you try a “model” that never flags anything. It’s right about all 167 good parts, an accuracy of 0.835, and it catches no defects at all. When one class is rare, accuracy mostly measures how common the other class is.
Moving the threshold
Suppose a missed defect costs far more than a second look at a good part. Lower the threshold to 0.3:
| Threshold | 0.5 | 0.3 |
|---|---|---|
| Defects caught (TP) | 15 | 28 |
| Defects missed (FN) | 18 | 5 |
| False alarms (FP) | 6 | 29 |
| Precision | 0.71 | 0.49 |
| Recall | 0.45 | 0.85 |
| F1 | 0.56 | 0.62 |
| Accuracy | 0.880 | 0.830 |
Now the model catches 28 of the 33 defects, at the price of 29 false alarms. F1 rises from 0.56 to 0.62. But accuracy falls, to 0.83, below the model that does nothing. By accuracy alone, you’d keep the threshold that misses more than half the defects. Which threshold is right depends on what each kind of mistake costs, and no single number can decide that for you.
One number for every threshold
The AUC, the area under the ROC curve, measures the scores themselves, before any threshold. It’s the chance that a randomly chosen defective part scores higher than a randomly chosen good one. We can count it directly: compare all 33 × 167 = 5,511 pairs, score a win as 1 and a tie as ½, and divide by 5,511. Here the AUC is 0.901. A model that guessed at random would get 0.5, and a perfect ranking 1.
Three pairs tie exactly, so ties matter. Every implementation scores a win as 2 points and a tie as 1, which keeps the total a whole number until the final division by 2 × 5,511.
What to watch for across languages
None of these implementations calls a ready-made metric. scikit-learn’s
roc_auc_score, R’s pROC package, and Perl’s Statistics::ROC compute
these, but they aren’t installed. The work is counting and dividing, and
both have traps.
Dividing one count by another is the first. In C and PostgreSQL, 7/2 is
3, so a ratio of counts needs a conversion before the division. Go, Swift,
OCaml, and Ada won’t divide an integer by a float at all until you convert
one of them. Common Lisp goes the other way and keeps exact fractions:
precision at 0.5 is 5/7, and the AUC is exactly 9931/11022 until it’s
printed.
Ties in the AUC are the second, because they depend on what “equal”
means. In Java, == on two boxed Double objects compares the objects,
not the values, so with boxed loop variables the 3 tied pairs would
quietly count as losses. APL’s = allows a tiny tolerance, so
0.3=0.1+0.2 is true there, which is harmless here because distinct
scores differ by at least 0.001. awk, COBOL, and Common Lisp sidestep the
question by reading each score as a whole number of thousandths.
Finally, watch how each language sorts a part into one of the four boxes.
Rust and Swift match on the pair (flagged, defective) and refuse to compile
if a case is missing. Perl and PHP score each AUC pair in one step:
<=> gives 1, 0, or −1, and adding 1 makes that the 2, 1, or 0 points
for a win, a tie, or a loss. Rust and Elixir can’t even call the
false-negative count fn, since that’s a keyword in both.
Data preview
| defective | score |
|---|---|
| 0 | 0.174 |
| 0 | 0.295 |
| 0 | 0.546 |
| 0 | 0.152 |
| 0 | 0.126 |
| 0 | 0.103 |
| 1 | 0.972 |
| 0 | 0.098 |
First 8 of 200 rows · Download data.csv
Expected results
Every implementation must print these, to within a relative tolerance of 10⁻⁴.
parts | 200 |
defective | 33 |
baseline_accuracy | 0.835000 |
tp_at_50 | 15 |
fp_at_50 | 6 |
fn_at_50 | 18 |
tn_at_50 | 161 |
accuracy_at_50 | 0.880000 |
precision_at_50 | 0.714286 |
recall_at_50 | 0.454545 |
f1_at_50 | 0.555556 |
tp_at_30 | 28 |
fp_at_30 | 29 |
fn_at_30 | 5 |
tn_at_30 | 138 |
accuracy_at_30 | 0.830000 |
precision_at_30 | 0.491228 |
recall_at_30 | 0.848485 |
f1_at_30 | 0.622222 |
auc | 0.901016 |
Implementations
"""Evaluation metrics for a binary classifier, using only the standard
library.
Reads each part's true label and the model's score, then reports the
confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
the accuracy of never flagging anything, and the area under the ROC curve.
"""
import csv
import sys
THRESHOLDS = (("50", 0.5), ("30", 0.3)) # name suffix, threshold
def read(path):
with open(path, newline="") as f:
return [(row["defective"] == "1", float(row["score"]))
for row in csv.DictReader(f)]
def report(parts, suffix, threshold):
# A part is flagged when its score reaches the threshold.
tp = fp = fn = tn = 0
for defective, score in parts:
flagged = score >= threshold
if flagged and defective:
tp += 1
elif flagged:
fp += 1
elif defective:
fn += 1
else:
tn += 1
accuracy = (tp + tn) / len(parts)
precision = tp / (tp + fp) # of the flagged parts, how many are defective
recall = tp / (tp + fn) # of the defective parts, how many are flagged
f1 = 2 * precision * recall / (precision + recall)
print(f"tp_at_{suffix} {tp}")
print(f"fp_at_{suffix} {fp}")
print(f"fn_at_{suffix} {fn}")
print(f"tn_at_{suffix} {tn}")
print(f"accuracy_at_{suffix} {accuracy:.6f}")
print(f"precision_at_{suffix} {precision:.6f}")
print(f"recall_at_{suffix} {recall:.6f}")
print(f"f1_at_{suffix} {f1:.6f}")
def auc(parts):
# The chance that a random defective part outscores a random good one.
# Compare every pair, scoring 2 for a win and 1 for a tie, so the count
# stays a whole number until the final division.
defective = [score for is_defective, score in parts if is_defective]
good = [score for is_defective, score in parts if not is_defective]
points = 0
for d in defective:
for g in good:
if d > g:
points += 2
elif d == g:
points += 1
return points / (2 * len(defective) * len(good))
def main():
parts = read(sys.argv[1])
defective = sum(is_defective for is_defective, _ in parts)
print(f"parts {len(parts)}")
print(f"defective {defective}")
# Never flagging anything is right about every good part.
print(f"baseline_accuracy {(len(parts) - defective) / len(parts):.6f}")
for suffix, threshold in THRESHOLDS:
report(parts, suffix, threshold)
print(f"auc {auc(parts):.6f}")
if __name__ == "__main__":
main()Each part becomes a pair of a bool and a float, and one loop per threshold sorts every part into one of four counts: true positives, false positives, false negatives, and true negatives. Every metric is a ratio of those counts.
The AUC compares every defective part with every good one, 33 times 167
pairs. A win scores 2 and a tie 1, so the total stays a whole number until
the last division.
scikit-learn’s roc_auc_score and classification_report do this, but
scikit-learn isn’t installed.
-- Evaluation metrics for a binary classifier, using only Ada's standard
-- library.
--
-- Reads each part's true label and the model's score, then reports the
-- confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
-- the accuracy of never flagging anything, and the area under the ROC
-- curve.
with Ada.Text_IO; use Ada.Text_IO;
with Ada.Long_Float_Text_IO; use Ada.Long_Float_Text_IO;
with Ada.Command_Line;
with Ada.Strings.Fixed; use Ada.Strings.Fixed;
with Ada.Containers.Vectors;
procedure Main is
type Part is record
Defective : Boolean;
Score : Long_Float;
end record;
package Part_Vectors is new Ada.Containers.Vectors (Positive, Part);
use Part_Vectors;
-- The text of field number Col (counting from 1) in a CSV line.
function Field (Line : String; Col : Positive) return String is
Start : Positive := Line'First;
begin
for I in 1 .. Col - 1 loop
Start := Index (Line (Start .. Line'Last), ",") + 1;
end loop;
declare
Stop : constant Natural := Index (Line (Start .. Line'Last), ",");
begin
return (if Stop = 0 then Line (Start .. Line'Last)
else Line (Start .. Stop - 1));
end;
end Field;
-- The number of the header field called Name.
function Column (Header, Name : String) return Positive is
begin
for C in 1 .. Ada.Strings.Fixed.Count (Header, ",") + 1 loop
if Field (Header, C) = Name then
return C;
end if;
end loop;
raise Constraint_Error with "column not found: " & Name;
end Column;
-- Ada never mixes Integer and Long_Float: each count is converted
-- before the division.
function Ratio (A, B : Natural) return Long_Float is
(Long_Float (A) / Long_Float (B));
procedure Put_Value (Name : String; X : Long_Float) is
begin
Put (Name & " ");
Put (X, Fore => 1, Aft => 6, Exp => 0);
New_Line;
end Put_Value;
-- 'Image puts a space before a non-negative number, which serves as
-- the space between the name and the count.
procedure Put_Count (Name : String; X : Natural) is
begin
Put_Line (Name & X'Image);
end Put_Count;
-- The confusion matrix and the metrics built from it, at one threshold.
procedure Report (Parts : Vector; Suffix : String; Threshold : Long_Float)
is
TP, FP, FN, TN : Natural := 0;
begin
for P of Parts loop
-- A part is flagged when its score reaches the threshold.
if P.Score >= Threshold then
if P.Defective then
TP := TP + 1;
else
FP := FP + 1;
end if;
elsif P.Defective then
FN := FN + 1;
else
TN := TN + 1;
end if;
end loop;
declare
Accuracy : constant Long_Float :=
Ratio (TP + TN, Natural (Parts.Length));
-- Of the flagged parts, how many are defective.
Precision : constant Long_Float := Ratio (TP, TP + FP);
-- Of the defective parts, how many are flagged.
Recall : constant Long_Float := Ratio (TP, TP + FN);
F1 : constant Long_Float :=
2.0 * Precision * Recall / (Precision + Recall);
begin
Put_Count ("tp_at_" & Suffix, TP);
Put_Count ("fp_at_" & Suffix, FP);
Put_Count ("fn_at_" & Suffix, FN);
Put_Count ("tn_at_" & Suffix, TN);
Put_Value ("accuracy_at_" & Suffix, Accuracy);
Put_Value ("precision_at_" & Suffix, Precision);
Put_Value ("recall_at_" & Suffix, Recall);
Put_Value ("f1_at_" & Suffix, F1);
end;
end Report;
-- The chance that a random defective part outscores a random good one.
-- Compare every pair, scoring 2 for a win and 1 for a tie, so the count
-- stays a whole number until the final division.
function AUC (Parts : Vector) return Long_Float is
Points, Defective, Good : Natural := 0;
begin
for D of Parts loop
if D.Defective then
Defective := Defective + 1;
for G of Parts loop
if not G.Defective then
if D.Score > G.Score then
Points := Points + 2;
elsif D.Score = G.Score then
Points := Points + 1;
end if;
end if;
end loop;
else
Good := Good + 1;
end if;
end loop;
return Ratio (Points, 2 * Defective * Good);
end AUC;
File : File_Type;
Parts : Vector;
begin
Open (File, In_File, Ada.Command_Line.Argument (1));
declare
Header : constant String := Get_Line (File);
Defective_C : constant Positive := Column (Header, "defective");
Score_C : constant Positive := Column (Header, "score");
begin
while not End_Of_File (File) loop
declare
Line : constant String := Get_Line (File);
begin
-- Part'(...) names the type of the aggregate. A Vector can
-- also be written as an aggregate in Ada 2022, so without it
-- the call to Append would be ambiguous.
Parts.Append
(Part'(Defective => Field (Line, Defective_C) = "1",
Score => Long_Float'Value (Field (Line, Score_C))));
end;
end loop;
end;
Close (File);
declare
N : constant Natural := Natural (Parts.Length);
Defective : Natural := 0;
begin
for P of Parts loop
if P.Defective then
Defective := Defective + 1;
end if;
end loop;
Put_Count ("parts", N);
Put_Count ("defective", Defective);
-- Never flagging anything is right about every good part.
Put_Value ("baseline_accuracy", Ratio (N - Defective, N));
Report (Parts, "50", 0.5);
Report (Parts, "30", 0.3);
Put_Value ("auc", AUC (Parts));
end;
end Main;Each part is a record with a Boolean label and a Long_Float score,
held in a Vector. One loop per threshold sorts every part into one of
four Natural counts, and a small Ratio function converts two counts to
Long_Float before dividing, since Ada never mixes the two types.
Appending a part needs Part'(...) around the record: in Ada 2022 a
Vector can be written as an aggregate too, so GNAT calls the bare form
ambiguous. The AUC loops over every defective and good pair, 33 times 167,
adding 2 for a win and 1 for a tie.
⍝ Evaluation metrics for a binary classifier, from scratch in GNU APL.
⍝ Reads each part's true label and the model's score, then reports the
⍝ confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
⍝ the accuracy of never flagging anything, and the area under the ROC curve.
⍝ APL reads right to left, with no operator precedence: 2×3+4 is 2×(3+4).
⍝ ⍝ starts a comment; ← assigns; ⍵ is a function's right argument.
∇NAME SAY N
⍝ Print "name value" for a whole number. ⍕ formats a number as text.
⎕←NAME,' ',⍕N
∇
∇NAME SAY6 X;T
⍝ Print "name value" with six decimal places.
T←0 6⍕|X ⍝ |X is the size of X; 0 6⍕ gives 6 decimals
T←(∨\T≠' ')/T ⍝ drop the leading blank ⍕ adds
T←((T[1]='.')/'0'),T ⍝ ⍕ writes .835 for 0.835, so put the 0 back
⎕←NAME,' ',((X<0)/'-'),T ⍝ sign as -, since APL's own minus is ¯
∇
⍝ Read the file. ⎕ARG is the command line; ↑¯1↑ takes its last item, the
⍝ data path. ⎕FIO[26] reads the whole file as one string.
TEXT←⎕FIO[26]↑¯1↑⎕ARG
LINES←(TEXT≠⎕UCS 10)⊂TEXT ⍝ ⊂ splits at the 0s of the mask: one item per line
⍝ Split each (¨) line at commas; ⊃ stacks them into a table.
ROWS←⊃{(⍵≠',')⊂⍵}¨LINES
⍝ A column by header name (⍳), minus the header (1↓).
COLUMN←{1↓ROWS[;ROWS[1;]⍳⊂⍵]}
DEFECTIVE←⍎¨COLUMN 'defective' ⍝ ⍎ turns text into a number: 1s and 0s
SCORE←⍎¨COLUMN 'score'
N←≢SCORE ⍝ ≢ counts the items
∇SUFFIX REPORT THRESHOLD;FLAGGED;TP;FP;FN;TN;PRECISION;RECALL
⍝ Sort every part into one of four counts at this threshold, then print
⍝ the counts and the ratios built from them. Each count is the sum (+/)
⍝ of a 1-or-0 mask: ∧ is and, ~ is not.
FLAGGED←SCORE≥THRESHOLD ⍝ flagged when the score reaches it
TP←+/FLAGGED∧DEFECTIVE
FP←+/FLAGGED∧~DEFECTIVE
FN←+/(~FLAGGED)∧DEFECTIVE
TN←+/(~FLAGGED)∧~DEFECTIVE
PRECISION←TP÷TP+FP ⍝ of the flagged, how many are defective
RECALL←TP÷TP+FN ⍝ of the defective, how many are flagged
('tp_at_',SUFFIX) SAY TP
('fp_at_',SUFFIX) SAY FP
('fn_at_',SUFFIX) SAY FN
('tn_at_',SUFFIX) SAY TN
('accuracy_at_',SUFFIX) SAY6 (TP+TN)÷N
('precision_at_',SUFFIX) SAY6 PRECISION
('recall_at_',SUFFIX) SAY6 RECALL
('f1_at_',SUFFIX) SAY6 (2×PRECISION×RECALL)÷PRECISION+RECALL
∇
'parts' SAY N
'defective' SAY +/DEFECTIVE
⍝ Never flagging anything is right about every good part.
'baseline_accuracy' SAY6 (N-+/DEFECTIVE)÷N
'50' REPORT 0.5
'30' REPORT 0.3
⍝ The AUC: the chance that a random defective part outscores a random good
⍝ one. / keeps the scores where the mask is 1. The outer product ∘.> compares
⍝ every defective score with every good score at once, giving a table of 1s
⍝ and 0s, one row per defective part and one column per good part; ∘.=
⍝ does the same for ties. A win scores 2 and a tie 1, and , flattens the
⍝ table so +/ can add it up, a whole number until the final division.
D←DEFECTIVE/SCORE
G←(~DEFECTIVE)/SCORE
POINTS←+/,(2×D∘.>G)+D∘.=G
'auc' SAY6 POINTS÷2×(≢D)×≢G
)OFFAPL needs no loops here. Each cell of the confusion matrix is the sum of a
mask of 1s and 0s, such as +/FLAGGED∧DEFECTIVE, and the ratios follow.
For the AUC, the outer product D∘.>G compares every defective score with
every good one at once, giving a 33 by 167 table of 1s and 0s, and D∘.=G
marks the ties. Doubling the wins, adding the ties, and summing the
flattened table gives the points. APL’s = is tolerant: ⎕CT defaults to
1E¯13, so 0.3=0.1+0.2 is 1. The scores here differ by at least 0.001, so
only true ties count.
# Evaluation metrics for a binary classifier, in POSIX awk.
#
# Reads each part's true label and the model's score, then reports the
# confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
# the accuracy of never flagging anything, and the area under the ROC curve.
# Rows are stored as they are read; the work happens in END.
BEGIN { FS = "," }
# The header line: find each column by name.
NR == 1 {
for (i = 1; i <= NF; i++) {
if ($i == "defective") c_label = i
if ($i == "score") c_score = i
}
if (!c_label || !c_score) {
print "missing a column (defective, score)" > "/dev/stderr"
failed = 1
exit 1
}
next
}
# Scores have three decimals, so we keep them as whole thousandths: 0.174
# becomes 174. Whole numbers compare exactly, so tied scores stay tied.
{
n++
label[n] = ($c_label == 1)
score[n] = int($c_score * 1000 + 0.5)
if (label[n]) n_defective++
}
# Thresholds are in thousandths too: 500 is 0.5. awk passes scalars by value,
# and the counts after the real parameters are locals.
function report(suffix, threshold, i, flagged, tp, fp, fn, tn,
accuracy, precision, recall, f1) {
# A part is flagged when its score reaches the threshold.
tp = fp = fn = tn = 0
for (i = 1; i <= n; i++) {
flagged = (score[i] >= threshold)
if (flagged && label[i]) tp++
else if (flagged) fp++
else if (label[i]) fn++
else tn++
}
accuracy = (tp + tn) / n
precision = tp / (tp + fp) # of the flagged parts, how many are defective
recall = tp / (tp + fn) # of the defective parts, how many are flagged
f1 = 2 * precision * recall / (precision + recall)
printf "tp_at_%s %d\n", suffix, tp
printf "fp_at_%s %d\n", suffix, fp
printf "fn_at_%s %d\n", suffix, fn
printf "tn_at_%s %d\n", suffix, tn
printf "accuracy_at_%s %.6f\n", suffix, accuracy
printf "precision_at_%s %.6f\n", suffix, precision
printf "recall_at_%s %.6f\n", suffix, recall
printf "f1_at_%s %.6f\n", suffix, f1
}
# The chance that a random defective part outscores a random good one.
# Compare every pair, scoring 2 for a win and 1 for a tie, so the count
# stays a whole number until the final division.
function auc( i, j, points) {
points = 0
for (i = 1; i <= n; i++) {
if (!label[i]) continue
for (j = 1; j <= n; j++) {
if (label[j]) continue
if (score[i] > score[j]) points += 2
else if (score[i] == score[j]) points += 1
}
}
return points / (2 * n_defective * (n - n_defective))
}
END {
if (failed) exit 1
printf "parts %d\n", n
printf "defective %d\n", n_defective
# Never flagging anything is right about every good part.
printf "baseline_accuracy %.6f\n", (n - n_defective) / n
report("50", 500)
report("30", 300)
printf "auc %.6f\n", auc()
}We store every row in two arrays and keep each score as whole
thousandths: 0.174 becomes 174, rounded with int(x * 1000 + 0.5). The
thresholds become 500 and 300, so every comparison is between whole
numbers and tied scores stay tied.
report takes the suffix and threshold, and its counts and ratios are
extra parameters, the only way to get local variables in awk. The AUC
loops over every pair of parts, skips any pair that isn’t one defective
and one good, and scores 2 for a win and 1 for a tie.
10 REM Evaluation metrics for a binary classifier, in classic BASIC.
20 REM
30 REM Reads each part's true label and the model's score, then reports
40 REM the confusion matrix, accuracy, precision, recall, and F1 at two
50 REM thresholds, the accuracy of never flagging anything, and the area
60 REM under the ROC curve.
70 REM
80 REM run.sh passes the data path in the DATA environment variable,
90 REM because bwbasic can't read command-line arguments.
100 DIM DF(1000), SC(1000), FL$(20) : REM defective (1 or 0) and score
110 GOSUB 1000 : REM read the parts into DF(1..N) and SC(1..N)
120 NP = 0
130 FOR I = 1 TO N
140 NP = NP + DF(I)
150 NEXT I
160 PRINT "parts"; N
170 PRINT "defective"; NP
180 REM Never flagging anything is right about every good part.
190 L$ = "baseline_accuracy" : Y = (N - NP) / N : GOSUB 3000
200 SF$ = "50" : TH = .5 : GOSUB 2000
210 SF$ = "30" : TH = .3 : GOSUB 2000
220 REM AUC: the chance that a random defective part outscores a random
230 REM good one. Compare every pair, scoring 2 for a win and 1 for a tie,
240 REM so the count stays a whole number until the final division.
250 PT = 0
260 FOR I = 1 TO N
270 FOR J = 1 TO N
280 IF DF(I) = 1 AND DF(J) = 0 AND SC(I) > SC(J) THEN PT = PT + 2
290 IF DF(I) = 1 AND DF(J) = 0 AND SC(I) = SC(J) THEN PT = PT + 1
300 NEXT J
310 NEXT I
320 L$ = "auc" : Y = PT / (2 * NP * (N - NP)) : GOSUB 3000
330 SYSTEM : REM without SYSTEM, bwbasic waits at its prompt
1000 REM Read every part's label and score.
1010 F$ = ENVIRON$("DATA")
1020 OPEN "I", #1, F$
1030 LINE INPUT #1, L$
1040 GOSUB 4000 : REM find which field of the header holds each column
1050 FOR J = 1 TO NF
1060 IF FL$(J) = "defective" THEN CD = J
1070 IF FL$(J) = "score" THEN CS = J
1080 NEXT J
1090 N = 0
1100 IF EOF(1) THEN 1150
1110 LINE INPUT #1, L$
1120 GOSUB 4000
1130 N = N + 1 : DF(N) = VAL(FL$(CD)) : SC(N) = VAL(FL$(CS))
1140 GOTO 1100
1150 CLOSE #1
1160 RETURN
2000 REM Count and score the parts at threshold TH, naming lines with SF$.
2010 REM A part is flagged when its score reaches the threshold.
2020 TP = 0 : FP = 0 : FN = 0 : TN = 0
2030 FOR I = 1 TO N
2040 IF SC(I) >= TH AND DF(I) = 1 THEN TP = TP + 1
2050 IF SC(I) >= TH AND DF(I) = 0 THEN FP = FP + 1
2060 IF SC(I) < TH AND DF(I) = 1 THEN FN = FN + 1
2070 IF SC(I) < TH AND DF(I) = 0 THEN TN = TN + 1
2080 NEXT I
2090 PR = TP / (TP + FP) : REM of the flagged parts, how many are defective
2100 RC = TP / (TP + FN) : REM of the defective parts, how many are flagged
2110 PRINT "tp_at_"; SF$; TP
2120 PRINT "fp_at_"; SF$; FP
2130 PRINT "fn_at_"; SF$; FN
2140 PRINT "tn_at_"; SF$; TN
2150 L$ = "accuracy_at_" + SF$ : Y = (TP + TN) / N : GOSUB 3000
2160 L$ = "precision_at_" + SF$ : Y = PR : GOSUB 3000
2170 L$ = "recall_at_" + SF$ : Y = RC : GOSUB 3000
2180 L$ = "f1_at_" + SF$ : Y = 2 * PR * RC / (PR + RC) : GOSUB 3000
2190 RETURN
3000 REM Print L$ and Y with six decimals.
3010 PRINT L$; " ";
3020 PRINT USING "#.######"; Y
3030 RETURN
4000 REM Split L$ at its commas into FL$(1) to FL$(NF).
4010 NF = 0 : P = 1
4020 Q = INSTR(P, L$, ",")
4030 NF = NF + 1
4040 IF Q = 0 THEN FL$(NF) = MID$(L$, P) : RETURN
4050 FL$(NF) = MID$(L$, P, Q - P) : P = Q + 1
4060 GOTO 4020The parts live in two fixed arrays, DF for the label and SC for the
score. A subroutine at line 2000 takes the threshold in TH and its name
in SF$, then counts true positives, false positives, false negatives,
and true negatives with four IF lines joined by AND, since bwbasic
mishandles an IF nested inside another.
The AUC loops over every pair of parts and keeps those with a defective first and a good second, scoring 2 for a win and 1 for a tie. That’s 40,000 checks to use 5,511 pairs, still instant.
/* Evaluation metrics for a binary classifier, using only the C standard
* library.
*
* Reads each part's true label and the model's score, then reports the
* confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
* the accuracy of never flagging anything, and the area under the ROC
* curve. */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MAX_PARTS 1000
#define MAX_LINE 256
static int defective[MAX_PARTS]; /* 1 for a defective part, 0 for a good one */
static double score[MAX_PARTS];
static int part_count;
/* Cut a line at its commas, in place, and return the number of fields.
* strtok replaces each comma with '\0', so fields[i] points into line. */
static int split(char *line, char *fields[], int max) {
int n = 0;
for (char *f = strtok(line, ",\r\n"); f && n < max; f = strtok(NULL, ",\r\n"))
fields[n++] = f;
return n;
}
/* Sort every part into one of four counts at this threshold, then print
* the counts and the ratios built from them. */
static void report(const char *suffix, double threshold) {
int tp = 0, fp = 0, fn = 0, tn = 0;
for (int i = 0; i < part_count; i++) {
int flagged = score[i] >= threshold; /* flagged when the score reaches it */
if (flagged && defective[i]) tp++;
else if (flagged) fp++;
else if (defective[i]) fn++;
else tn++;
}
/* Casting one side to double makes the division a real one; int / int
* would throw away the fraction. */
double accuracy = (double)(tp + tn) / part_count;
double precision = (double)tp / (tp + fp); /* of the flagged, how many defective */
double recall = (double)tp / (tp + fn); /* of the defective, how many flagged */
double f1 = 2 * precision * recall / (precision + recall);
printf("tp_at_%s %d\n", suffix, tp);
printf("fp_at_%s %d\n", suffix, fp);
printf("fn_at_%s %d\n", suffix, fn);
printf("tn_at_%s %d\n", suffix, tn);
printf("accuracy_at_%s %.6f\n", suffix, accuracy);
printf("precision_at_%s %.6f\n", suffix, precision);
printf("recall_at_%s %.6f\n", suffix, recall);
printf("f1_at_%s %.6f\n", suffix, f1);
}
/* The chance that a random defective part outscores a random good one.
* Compare every pair, scoring 2 for a win and 1 for a tie, so the count
* stays a whole number until the final division. */
static double auc(void) {
long points = 0, pairs = 0;
for (int d = 0; d < part_count; d++) {
if (!defective[d]) continue;
for (int g = 0; g < part_count; g++) {
if (defective[g]) continue;
pairs++;
if (score[d] > score[g]) points += 2;
else if (score[d] == score[g]) points += 1; /* same text, same double */
}
}
return (double)points / (2 * pairs);
}
int main(int argc, char **argv) {
if (argc < 2) { fprintf(stderr, "usage: %s data.csv\n", argv[0]); return 1; }
FILE *file = fopen(argv[1], "r");
if (!file) { perror(argv[1]); return 1; }
char line[MAX_LINE], *fields[4];
if (!fgets(line, sizeof line, file)) { fprintf(stderr, "empty file\n"); return 1; }
int defective_count = 0;
while (fgets(line, sizeof line, file) && part_count < MAX_PARTS) {
if (split(line, fields, 4) != 2) continue;
defective[part_count] = strcmp(fields[0], "1") == 0;
score[part_count] = strtod(fields[1], NULL);
defective_count += defective[part_count];
part_count++;
}
fclose(file);
printf("parts %d\n", part_count);
printf("defective %d\n", defective_count);
/* Never flagging anything is right about every good part. */
printf("baseline_accuracy %.6f\n", (double)(part_count - defective_count) / part_count);
report("50", 0.5);
report("30", 0.3);
printf("auc %.6f\n", auc());
return 0;
}We read the labels and scores into two parallel arrays. report makes
one pass per threshold, sorting each part into true positives, false
positives, false negatives, or true negatives. Each metric is a ratio of
those four int counts, and we cast one side to double before dividing,
since int / int in C drops the fraction.
The AUC uses two nested loops, skipping the wrong label in
each, so every defective part meets every good one. A win adds 2 points and
a tie 1, the total stays a long, and we divide only once, at the end.
Scores that are equal text in the file parse to equal doubles, so ==
catches the 3 tied pairs.
// Evaluation metrics for a binary classifier, using only the C++ standard
// library.
//
// Reads each part's true label and the model's score, then reports the
// confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
// the accuracy of never flagging anything, and the area under the ROC curve.
#include <algorithm>
#include <fstream>
#include <print>
#include <string>
#include <string_view>
#include <vector>
struct Part {
bool defective;
double score;
};
// Sort every part into one of four counts at this threshold, then print the
// counts and the ratios built from them.
void report(const std::vector<Part>& parts, std::string_view suffix, double threshold) {
int tp = 0, fp = 0, fn = 0, tn = 0;
for (const auto& [defective, score] : parts) { // structured binding unpacks a Part
bool flagged = score >= threshold; // flagged when the score reaches it
if (flagged && defective) {
tp++;
} else if (flagged) {
fp++;
} else if (defective) {
fn++;
} else {
tn++;
}
}
double accuracy = static_cast<double>(tp + tn) / parts.size();
double precision = static_cast<double>(tp) / (tp + fp); // of the flagged, how many defective
double recall = static_cast<double>(tp) / (tp + fn); // of the defective, how many flagged
double f1 = 2 * precision * recall / (precision + recall);
std::println("tp_at_{} {}", suffix, tp);
std::println("fp_at_{} {}", suffix, fp);
std::println("fn_at_{} {}", suffix, fn);
std::println("tn_at_{} {}", suffix, tn);
std::println("accuracy_at_{} {:.6f}", suffix, accuracy);
std::println("precision_at_{} {:.6f}", suffix, precision);
std::println("recall_at_{} {:.6f}", suffix, recall);
std::println("f1_at_{} {:.6f}", suffix, f1);
}
// The chance that a random defective part outscores a random good one.
// Compare every pair, scoring 2 for a win and 1 for a tie, so the count
// stays a whole number until the final division.
double auc(const std::vector<Part>& parts) {
std::vector<double> defective, good;
for (const auto& part : parts) {
(part.defective ? defective : good).push_back(part.score);
}
long points = 0;
for (double d : defective) {
for (double g : good) {
if (d > g) {
points += 2;
} else if (d == g) { // equal text in the file parses to equal doubles
points += 1;
}
}
}
return static_cast<double>(points) / (2.0 * defective.size() * good.size());
}
int main(int argc, char* argv[]) {
if (argc != 2) {
std::println(stderr, "usage: program data.csv");
return 1;
}
// Read defective,score lines.
std::ifstream file(argv[1]);
std::string line;
std::getline(file, line); // skip the header
std::vector<Part> parts;
while (std::getline(file, line)) {
auto comma = line.find(',');
if (comma == std::string::npos) {
continue;
}
parts.push_back({line.substr(0, comma) == "1", std::stod(line.substr(comma + 1))});
}
// count_if counts the elements the lambda says yes to.
auto defective = std::ranges::count_if(parts, [](const Part& p) { return p.defective; });
std::println("parts {}", parts.size());
std::println("defective {}", defective);
// Never flagging anything is right about every good part.
std::println("baseline_accuracy {:.6f}",
static_cast<double>(parts.size() - defective) / parts.size());
report(parts, "50", 0.5);
report(parts, "30", 0.3);
std::println("auc {:.6f}", auc(parts));
}Each row becomes a Part struct holding a bool and a double. report
loops over the parts with a structured binding, which unpacks each struct
into defective and score, and sorts every part into one of four counts.
The metrics are ratios of those counts, cast to double before dividing.
For the AUC we split the scores into a defective vector and a good vector,
then compare every pair in two nested loops, 2 points for a win and 1 for a
tie, in a long. std::ranges::count_if counts the defective parts, and
every metric is written out by hand rather than called.
*> Evaluation metrics for a binary classifier, in COBOL.
*>
*> Reads each part's true label and the model's score, then
*> reports the confusion matrix, accuracy, precision, recall, and
*> F1 at two thresholds, the accuracy of never flagging anything,
*> and the area under the ROC curve.
IDENTIFICATION DIVISION.
PROGRAM-ID. EVALUATION-METRICS.
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT DATA-FILE ASSIGN TO DYNAMIC WS-FILE-NAME
ORGANIZATION IS LINE SEQUENTIAL.
DATA DIVISION.
FILE SECTION.
FD DATA-FILE.
01 DATA-LINE PIC X(200).
WORKING-STORAGE SECTION.
01 WS-FILE-NAME PIC X(256).
01 WS-EOF PIC X VALUE "N".
88 END-OF-FILE VALUE "Y".
01 WS-LABEL-TEXT PIC X(40).
01 WS-SCORE-TEXT PIC X(40).
*> Every part's label and score, in file order. Scores have three
*> decimals, so we keep them as whole thousandths: 0.174 is 174.
*> Every comparison below is between exact integers.
01 WS-PART-COUNT PIC 9(4) VALUE 0.
01 WS-DEFECTIVE-COUNT PIC 9(4) VALUE 0.
01 WS-PARTS.
05 WS-PART OCCURS 1000 TIMES.
10 WS-DEFECTIVE PIC 9.
10 WS-SCORE PIC 9(4).
*> The threshold being reported, also in thousandths, and the
*> name suffix that goes with it.
01 WS-THRESHOLD PIC 9(4).
01 WS-SUFFIX PIC XX.
01 WS-TP PIC 9(4).
01 WS-FP PIC 9(4).
01 WS-FN PIC 9(4).
01 WS-TN PIC 9(4).
01 WS-PRECISION PIC 9V9(15).
01 WS-RECALL PIC 9V9(15).
01 WS-POINTS PIC 9(9) VALUE 0.
01 WS-PAIRS PIC 9(9) VALUE 0.
01 I PIC 9(4).
01 J PIC 9(4).
01 WS-RESULT PIC S9(9)V9(12).
01 WS-RESULT-OUT PIC -(9)9.9(6).
01 WS-COUNT-OUT PIC Z(8)9.
PROCEDURE DIVISION.
MAIN.
ACCEPT WS-FILE-NAME FROM ARGUMENT-VALUE
OPEN INPUT DATA-FILE
*> The first line is the header, defective,score; skip it.
READ DATA-FILE
AT END SET END-OF-FILE TO TRUE
END-READ
PERFORM UNTIL END-OF-FILE
READ DATA-FILE
AT END SET END-OF-FILE TO TRUE
NOT AT END PERFORM ADD-PART
END-READ
END-PERFORM
CLOSE DATA-FILE
MOVE WS-PART-COUNT TO WS-COUNT-OUT
DISPLAY "parts " FUNCTION TRIM(WS-COUNT-OUT)
MOVE WS-DEFECTIVE-COUNT TO WS-COUNT-OUT
DISPLAY "defective " FUNCTION TRIM(WS-COUNT-OUT)
*> Never flagging anything is right about every good part.
COMPUTE WS-RESULT ROUNDED =
(WS-PART-COUNT - WS-DEFECTIVE-COUNT) / WS-PART-COUNT
COMPUTE WS-RESULT-OUT ROUNDED = WS-RESULT
DISPLAY "baseline_accuracy " FUNCTION TRIM(WS-RESULT-OUT)
MOVE 500 TO WS-THRESHOLD
MOVE "50" TO WS-SUFFIX
PERFORM REPORT-THRESHOLD
MOVE 300 TO WS-THRESHOLD
MOVE "30" TO WS-SUFFIX
PERFORM REPORT-THRESHOLD
PERFORM AREA-UNDER-CURVE
STOP RUN.
*> Store one defective,score line.
ADD-PART.
MOVE SPACES TO WS-LABEL-TEXT WS-SCORE-TEXT
UNSTRING DATA-LINE DELIMITED BY ","
INTO WS-LABEL-TEXT WS-SCORE-TEXT
END-UNSTRING
IF WS-LABEL-TEXT = SPACES
EXIT PARAGRAPH
END-IF
ADD 1 TO WS-PART-COUNT
COMPUTE WS-DEFECTIVE(WS-PART-COUNT) =
FUNCTION NUMVAL(WS-LABEL-TEXT)
*> NUMVAL reads "0.174" as 0.174; times 1000 is exactly 174.
COMPUTE WS-SCORE(WS-PART-COUNT) =
FUNCTION NUMVAL(WS-SCORE-TEXT) * 1000
ADD WS-DEFECTIVE(WS-PART-COUNT) TO WS-DEFECTIVE-COUNT.
*> Sort every part into one of four counts at WS-THRESHOLD, then
*> print the counts and the ratios built from them. A part is
*> flagged when its score reaches the threshold.
REPORT-THRESHOLD.
MOVE 0 TO WS-TP WS-FP WS-FN WS-TN
PERFORM VARYING I FROM 1 BY 1 UNTIL I > WS-PART-COUNT
EVALUATE WS-SCORE(I) >= WS-THRESHOLD
ALSO WS-DEFECTIVE(I) = 1
WHEN TRUE ALSO TRUE ADD 1 TO WS-TP
WHEN TRUE ALSO FALSE ADD 1 TO WS-FP
WHEN FALSE ALSO TRUE ADD 1 TO WS-FN
WHEN FALSE ALSO FALSE ADD 1 TO WS-TN
END-EVALUATE
END-PERFORM
MOVE WS-TP TO WS-COUNT-OUT
DISPLAY "tp_at_" WS-SUFFIX " " FUNCTION TRIM(WS-COUNT-OUT)
MOVE WS-FP TO WS-COUNT-OUT
DISPLAY "fp_at_" WS-SUFFIX " " FUNCTION TRIM(WS-COUNT-OUT)
MOVE WS-FN TO WS-COUNT-OUT
DISPLAY "fn_at_" WS-SUFFIX " " FUNCTION TRIM(WS-COUNT-OUT)
MOVE WS-TN TO WS-COUNT-OUT
DISPLAY "tn_at_" WS-SUFFIX " " FUNCTION TRIM(WS-COUNT-OUT)
COMPUTE WS-RESULT ROUNDED = (WS-TP + WS-TN) / WS-PART-COUNT
COMPUTE WS-RESULT-OUT ROUNDED = WS-RESULT
DISPLAY "accuracy_at_" WS-SUFFIX " "
FUNCTION TRIM(WS-RESULT-OUT)
*> Precision: of the flagged parts, how many are defective.
*> Recall: of the defective parts, how many are flagged.
COMPUTE WS-PRECISION ROUNDED = WS-TP / (WS-TP + WS-FP)
COMPUTE WS-RECALL ROUNDED = WS-TP / (WS-TP + WS-FN)
COMPUTE WS-RESULT-OUT ROUNDED = WS-PRECISION
DISPLAY "precision_at_" WS-SUFFIX " "
FUNCTION TRIM(WS-RESULT-OUT)
COMPUTE WS-RESULT-OUT ROUNDED = WS-RECALL
DISPLAY "recall_at_" WS-SUFFIX " "
FUNCTION TRIM(WS-RESULT-OUT)
COMPUTE WS-RESULT ROUNDED = 2 * WS-PRECISION * WS-RECALL
/ (WS-PRECISION + WS-RECALL)
COMPUTE WS-RESULT-OUT ROUNDED = WS-RESULT
DISPLAY "f1_at_" WS-SUFFIX " " FUNCTION TRIM(WS-RESULT-OUT).
*> The chance that a random defective part outscores a random
*> good one. Compare every pair, scoring 2 for a win and 1 for a
*> tie, so the count stays a whole number until the final
*> division.
AREA-UNDER-CURVE.
PERFORM VARYING I FROM 1 BY 1 UNTIL I > WS-PART-COUNT
IF WS-DEFECTIVE(I) = 1
PERFORM VARYING J FROM 1 BY 1
UNTIL J > WS-PART-COUNT
IF WS-DEFECTIVE(J) = 0
ADD 1 TO WS-PAIRS
EVALUATE TRUE
WHEN WS-SCORE(I) > WS-SCORE(J)
ADD 2 TO WS-POINTS
WHEN WS-SCORE(I) = WS-SCORE(J)
ADD 1 TO WS-POINTS
END-EVALUATE
END-IF
END-PERFORM
END-IF
END-PERFORM
COMPUTE WS-RESULT ROUNDED = WS-POINTS / (2 * WS-PAIRS)
COMPUTE WS-RESULT-OUT ROUNDED = WS-RESULT
DISPLAY "auc " FUNCTION TRIM(WS-RESULT-OUT).We keep each score as whole thousandths in a PIC 9(4) field: NUMVAL
reads “0.174”, and multiplying by 1000 stores exactly 174. The thresholds
become 500 and 300, so every comparison is between exact integers, and the
3 tied pairs compare equal.
REPORT-THRESHOLD sorts the parts with EVALUATE ... ALSO ..., which tests
two conditions at once, flagged and defective, with one WHEN line per
cell of the confusion matrix. The AUC is two nested PERFORM VARYING loops
over the table, adding 2 points for a win and 1 for a tie. The ratios are
fixed-point decimals, rounded to 6 places for printing.
;;;; Evaluation metrics for a binary classifier, using only standard
;;;; Common Lisp.
;;;;
;;;; Reads each part's true label and the model's score, then reports the
;;;; confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
;;;; the accuracy of never flagging anything, and the area under the ROC
;;;; curve.
(defun split-on-comma (line)
"The fields of LINE, split at every comma."
(loop for start = 0 then (1+ end)
for end = (position #\, line :start start)
collect (subseq line start end)
while end))
(defun thousandths (text)
"A score such as \"0.174\" as the whole number of thousandths, 174.
parse-integer reads digits only; it never runs code, as READ could."
(parse-integer (remove #\. text)))
(defun read-parts (path)
"The rows of the CSV file at PATH, as (defective . score) pairs in file
order. Scores are whole thousandths, so equal scores stay exactly equal."
(with-open-file (in path)
(let* ((header (split-on-comma (read-line in)))
(defective-at (or (position "defective" header :test #'string=)
(error "no column defective")))
(score-at (or (position "score" header :test #'string=)
(error "no column score"))))
(loop for line = (read-line in nil)
while line
unless (string= line "")
collect (let ((fields (split-on-comma line)))
(cons (string= (nth defective-at fields) "1")
(thousandths (nth score-at fields))))))))
;;; ~d prints an integer, ~,6f a number with six decimals, ~a a string, and
;;; ~% a newline. Dividing two integers gives an exact fraction, like 5/7,
;;; so every ratio stays exact until (float x 1d0) makes it a double.
(defun print-metrics (parts suffix threshold)
"Print the confusion matrix and its metrics when parts scoring THRESHOLD
thousandths or more are flagged."
(let ((tp 0) (fp 0) (fn 0) (tn 0))
(loop for (defective . score) in parts
for flagged = (>= score threshold)
do (cond ((and flagged defective) (incf tp))
(flagged (incf fp))
(defective (incf fn))
(t (incf tn))))
(let* ((accuracy (/ (+ tp tn) (length parts)))
(precision (/ tp (+ tp fp))) ; of the flagged parts, how many are defective
(recall (/ tp (+ tp fn))) ; of the defective parts, how many are flagged
(f1 (/ (* 2 precision recall) (+ precision recall))))
(format t "tp_at_~a ~d~%" suffix tp)
(format t "fp_at_~a ~d~%" suffix fp)
(format t "fn_at_~a ~d~%" suffix fn)
(format t "tn_at_~a ~d~%" suffix tn)
(format t "accuracy_at_~a ~,6f~%" suffix (float accuracy 1d0))
(format t "precision_at_~a ~,6f~%" suffix (float precision 1d0))
(format t "recall_at_~a ~,6f~%" suffix (float recall 1d0))
(format t "f1_at_~a ~,6f~%" suffix (float f1 1d0)))))
(defun auc (parts)
"The chance that a random defective part outscores a random good one.
Every pair scores 2 for a win and 1 for a tie, so the count stays a whole
number until the final division."
(let ((defective (loop for (d . s) in parts when d collect s))
(good (loop for (d . s) in parts unless d collect s)))
(/ (loop for d in defective
sum (loop for g in good
sum (cond ((> d g) 2) ((= d g) 1) (t 0))))
(* 2 (length defective) (length good)))))
(let* ((parts (read-parts (second sb-ext:*posix-argv*)))
(n (length parts))
(defective (count-if #'car parts)))
(format t "parts ~d~%" n)
(format t "defective ~d~%" defective)
;; Never flagging anything is right about every good part.
(format t "baseline_accuracy ~,6f~%" (float (/ (- n defective) n) 1d0))
;; The thresholds 0.5 and 0.3, in thousandths.
(print-metrics parts "50" 500)
(print-metrics parts "30" 300)
(format t "auc ~,6f~%" (float (auc parts) 1d0)))We read each score as whole thousandths, so “0.174” becomes 174:
parse-integer on the digits with the point removed. The thresholds become
500 and 300, and equal scores stay exactly equal.
Dividing two integers in Common Lisp gives an exact fraction, so precision
at 0.5 is 5/7, F1 is 5/9, and the AUC is 9931/11022 until float turns
each into a double for printing. The function is print-metrics, not
report: SBCL’s default package imports report from its profiler, and
redefining it trips a package lock.
# Evaluation metrics for a binary classifier, using only Elixir's standard
# library.
#
# Reads each part's true label and the model's score, then reports the
# confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
# the accuracy of never flagging anything, and the area under the ROC curve.
defmodule EvaluationMetrics do
# Every part as {defective, score}, with each column found by its header
# name. String.to_float needs a decimal point, which every score has.
def read_parts(path) do
[header | rows] =
path
|> File.read!()
|> String.split("\n", trim: true)
|> Enum.map(&String.split(&1, ","))
for fields <- rows do
row = header |> Enum.zip(fields) |> Map.new()
{row["defective"] == "1", String.to_float(row["score"])}
end
end
# / always returns a float in Elixir, even for 6 / 3.
def format(x), do: :erlang.float_to_binary(x, decimals: 6)
def report(parts, suffix, threshold) do
# A part is flagged when its score reaches the threshold. frequencies
# counts each {flagged, defective} pair; a missing pair counts as 0.
counts =
Enum.frequencies(for {defective, score} <- parts, do: {score >= threshold, defective})
tp = Map.get(counts, {true, true}, 0)
fp = Map.get(counts, {true, false}, 0)
# fn starts an anonymous function in Elixir, so it can't be a name.
fn_ = Map.get(counts, {false, true}, 0)
tn = Map.get(counts, {false, false}, 0)
accuracy = (tp + tn) / length(parts)
# Of the flagged parts, how many are defective.
precision = tp / (tp + fp)
# Of the defective parts, how many are flagged.
recall = tp / (tp + fn_)
f1 = 2 * precision * recall / (precision + recall)
IO.puts("tp_at_#{suffix} #{tp}")
IO.puts("fp_at_#{suffix} #{fp}")
IO.puts("fn_at_#{suffix} #{fn_}")
IO.puts("tn_at_#{suffix} #{tn}")
IO.puts("accuracy_at_#{suffix} #{format(accuracy)}")
IO.puts("precision_at_#{suffix} #{format(precision)}")
IO.puts("recall_at_#{suffix} #{format(recall)}")
IO.puts("f1_at_#{suffix} #{format(f1)}")
end
# The chance that a random defective part outscores a random good one.
# Compare every pair, scoring 2 for a win and 1 for a tie, so the count
# stays a whole number until the final division.
def auc(parts) do
defective = for {true, score} <- parts, do: score
good = for {false, score} <- parts, do: score
points =
Enum.sum(
for d <- defective, g <- good do
cond do
d > g -> 2
d == g -> 1
true -> 0
end
end
)
points / (2 * length(defective) * length(good))
end
end
[path] = System.argv()
parts = EvaluationMetrics.read_parts(path)
n = length(parts)
defective = Enum.count(parts, fn {defective, _score} -> defective end)
IO.puts("parts #{n}")
IO.puts("defective #{defective}")
# Never flagging anything is right about every good part.
IO.puts("baseline_accuracy #{EvaluationMetrics.format((n - defective) / n)}")
EvaluationMetrics.report(parts, "50", 0.5)
EvaluationMetrics.report(parts, "30", 0.3)
IO.puts("auc #{EvaluationMetrics.format(EvaluationMetrics.auc(parts))}")Each part becomes a {defective, score} tuple. At each threshold,
Enum.frequencies counts the {flagged, defective} pairs, and the four
cells of the confusion matrix are four lookups, with 0 for a missing pair.
The false-negative count is named fn_, because fn starts an anonymous
function and can’t be a variable.
The AUC is a comprehension over every defective and good score, 33 times
167 pairs, and a cond gives 2 for a win, 1 for a tie, and 0 for a loss.
Enum.sum adds them, and / gives a float only at the end.
! Evaluation metrics for a binary classifier, using only standard Fortran.
!
! Reads each part's true label and the model's score, then reports the
! confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
! the accuracy of never flagging anything, and the area under the ROC curve.
program evaluation_metrics
implicit none
integer, parameter :: dp = kind(1.0d0) ! double precision
logical, allocatable :: defective(:)
real(dp), allocatable :: score(:)
integer :: n, positives
call read_parts(defective, score)
n = size(score)
positives = count(defective) ! count() counts the .true. elements
print "(a, 1x, i0)", "parts", n
print "(a, 1x, i0)", "defective", positives
! Never flagging anything is right about every good part.
call say("baseline_accuracy", real(n - positives, dp) / n)
call report("50", 0.5_dp)
call report("30", 0.3_dp)
call say("auc", auc())
contains
! The confusion matrix and the metrics built from it, at one threshold.
subroutine report(suffix, threshold)
character(len=*), intent(in) :: suffix
real(dp), intent(in) :: threshold
logical :: flagged(n)
integer :: tp, fp, fn, tn
real(dp) :: accuracy, precision, recall, f1
! A part is flagged when its score reaches the threshold. The comparison
! runs on the whole array at once, giving one .true. or .false. per part.
flagged = score >= threshold
tp = count(flagged .and. defective)
fp = count(flagged .and. .not. defective)
fn = count(.not. flagged .and. defective)
tn = count(.not. flagged .and. .not. defective)
accuracy = real(tp + tn, dp) / n
precision = real(tp, dp) / (tp + fp) ! of the flagged parts, how many are defective
recall = real(tp, dp) / (tp + fn) ! of the defective parts, how many are flagged
f1 = 2 * precision * recall / (precision + recall)
print "(a, 1x, i0)", "tp_at_" // suffix, tp
print "(a, 1x, i0)", "fp_at_" // suffix, fp
print "(a, 1x, i0)", "fn_at_" // suffix, fn
print "(a, 1x, i0)", "tn_at_" // suffix, tn
call say("accuracy_at_" // suffix, accuracy)
call say("precision_at_" // suffix, precision)
call say("recall_at_" // suffix, recall)
call say("f1_at_" // suffix, f1)
end subroutine report
! The chance that a random defective part outscores a random good one.
! Compare every pair, scoring 2 for a win and 1 for a tie, so the count
! stays a whole number until the final division.
real(dp) function auc()
real(dp), allocatable :: good(:)
integer :: points, i
good = pack(score, .not. defective) ! pack keeps the elements where the mask is .true.
points = 0
do i = 1, n
if (defective(i)) then
! One defective part against every good part at once.
points = points + 2 * count(score(i) > good) + count(score(i) == good)
end if
end do
auc = real(points, dp) / (2 * positives * size(good))
end function auc
subroutine read_parts(defective, score)
logical, allocatable, intent(out) :: defective(:)
real(dp), allocatable, intent(out) :: score(:)
character(len=256) :: path, line, text
integer :: unit, status, defective_col, score_col
real(dp) :: x
call get_command_argument(1, path)
open(newunit=unit, file=path, status="old", action="read")
read(unit, "(a)") line
defective_col = field_index(line, "defective")
score_col = field_index(line, "score")
allocate(defective(0), score(0))
do
read(unit, "(a)", iostat=status) line
if (status /= 0) exit
if (len_trim(line) == 0) cycle
! An internal read parses a number. It must read from a variable,
! not straight from the result of field(), so we copy into text.
text = field(line, score_col)
read(text, *) x
score = [score, x] ! grows the array by one
defective = [defective, field(line, defective_col) == "1"]
end do
close(unit)
end subroutine read_parts
! The position of a name in a comma-separated header line.
integer function field_index(header, name)
character(len=*), intent(in) :: header, name
integer :: i, fields
fields = 1
do i = 1, len_trim(header)
if (header(i:i) == ",") fields = fields + 1
end do
do field_index = 1, fields
if (field(header, field_index) == name) return
end do
error stop "column not found: " // name
end function field_index
! The k-th comma-separated field of a line.
function field(line, k) result(text)
character(len=*), intent(in) :: line
integer, intent(in) :: k
character(len=:), allocatable :: text
integer :: start, i, comma
start = 1
do i = 1, k - 1
start = start + index(line(start:), ",")
end do
comma = index(line(start:), ",")
if (comma == 0) then
text = trim(line(start:))
else
text = line(start:start + comma - 2)
end if
end function field
! Print "name value" with six decimals. The f0.6 format drops the zero
! before the point (0.5 prints as .500000), so we put it back.
subroutine say(name, x)
character(len=*), intent(in) :: name
real(dp), intent(in) :: x
character(len=40) :: text
write(text, "(f0.6)") x
if (text(1:1) == ".") text = "0" // text
if (text(1:2) == "-.") text = "-0" // text(2:)
print "(a, 1x, a)", name, trim(text)
end subroutine say
end program evaluation_metricsWe keep two arrays, one of logical labels and one of scores, and work on
whole arrays at once. score >= threshold gives one .true. or .false.
per part, and count of a mask such as flagged .and. defective gives
each cell of the confusion matrix with no loop.
For the AUC, pack keeps the good scores, and each defective score is
compared against all of them in one expression: twice the count of good
scores below it, plus the count equal to it. That adds up 33 times 167
pairs while the total stays an integer.
// Evaluation metrics for a binary classifier, using only the standard
// library.
//
// Reads each part's true label and the model's score, then reports the
// confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
// the accuracy of never flagging anything, and the area under the ROC curve.
package main
import (
"encoding/csv"
"fmt"
"os"
"strconv"
)
type part struct {
defective bool
score float64
}
func read(path string) ([]part, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
records, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, err
}
// Find each column by name in the header.
iLabel, iScore := -1, -1
for i, name := range records[0] {
switch name {
case "defective":
iLabel = i
case "score":
iScore = i
}
}
if iLabel < 0 || iScore < 0 {
return nil, fmt.Errorf("missing a column (defective, score)")
}
var parts []part
for _, record := range records[1:] {
score, err := strconv.ParseFloat(record[iScore], 64)
if err != nil {
return nil, err
}
parts = append(parts, part{record[iLabel] == "1", score})
}
return parts, nil
}
func report(parts []part, suffix string, threshold float64) {
// A part is flagged when its score reaches the threshold.
var tp, fp, fn, tn int
for _, p := range parts {
flagged := p.score >= threshold
switch {
case flagged && p.defective:
tp++
case flagged:
fp++
case p.defective:
fn++
default:
tn++
}
}
// Go never converts int to float64 on its own, so each ratio says so.
accuracy := float64(tp+tn) / float64(len(parts))
precision := float64(tp) / float64(tp+fp) // of the flagged parts, how many are defective
recall := float64(tp) / float64(tp+fn) // of the defective parts, how many are flagged
f1 := 2 * precision * recall / (precision + recall)
fmt.Printf("tp_at_%s %d\n", suffix, tp)
fmt.Printf("fp_at_%s %d\n", suffix, fp)
fmt.Printf("fn_at_%s %d\n", suffix, fn)
fmt.Printf("tn_at_%s %d\n", suffix, tn)
fmt.Printf("accuracy_at_%s %.6f\n", suffix, accuracy)
fmt.Printf("precision_at_%s %.6f\n", suffix, precision)
fmt.Printf("recall_at_%s %.6f\n", suffix, recall)
fmt.Printf("f1_at_%s %.6f\n", suffix, f1)
}
// auc is the chance that a random defective part outscores a random good
// one. Compare every pair, scoring 2 for a win and 1 for a tie, so the
// count stays a whole number until the final division.
func auc(parts []part) float64 {
var defective, good []float64
for _, p := range parts {
if p.defective {
defective = append(defective, p.score)
} else {
good = append(good, p.score)
}
}
points := 0
for _, d := range defective {
for _, g := range good {
if d > g {
points += 2
} else if d == g {
points++
}
}
}
return float64(points) / float64(2*len(defective)*len(good))
}
func main() {
parts, err := read(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defective := 0
for _, p := range parts {
if p.defective {
defective++
}
}
fmt.Printf("parts %d\n", len(parts))
fmt.Printf("defective %d\n", defective)
// Never flagging anything is right about every good part.
fmt.Printf("baseline_accuracy %.6f\n", float64(len(parts)-defective)/float64(len(parts)))
report(parts, "50", 0.5)
report(parts, "30", 0.3)
fmt.Printf("auc %.6f\n", auc(parts))
}encoding/csv reads the file, and each row becomes a part struct with a
bool and a float64. A tagless switch sorts every part into true
positives, false positives, false negatives, and true negatives, taking the
first case that holds.
Go never mixes int and float64 in arithmetic, so dividing an int by
a float64 doesn’t compile, and every ratio wraps its counts in
float64(...). The AUC loops over all 33 times 167 pairs, scoring 2 for a
win and 1 for a tie in an int.
-- Evaluation metrics for a binary classifier, using only Haskell's base
-- library.
--
-- Reads each part's true label and the model's score, then reports the
-- confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
-- the accuracy of never flagging anything, and the area under the ROC curve.
import Data.List (elemIndex)
import System.Environment (getArgs)
import Text.Printf (printf)
-- Split a line at every comma. break returns the text before the first comma
-- and the rest, which still starts with the comma.
splitOnComma :: String -> [String]
splitOnComma line = case break (== ',') line of
(field, ',' : rest) -> field : splitOnComma rest
(field, _) -> [field]
-- The position of a column in the header, or a stop with a message.
columnIndex :: [String] -> String -> Int
columnIndex header name = case elemIndex name header of
Just index -> index
Nothing -> error ("no column " ++ name)
-- Every part as (defective, score), in file order.
readParts :: String -> [(Bool, Double)]
readParts text = case lines text of
[] -> error "empty file"
headerLine : rows ->
[ (field "defective" == "1", read (field "score"))
| row <- rows
, not (null row)
, let fields = splitOnComma row
field name = fields !! columnIndex header name
]
where
header = splitOnComma headerLine
-- fromIntegral turns an Int into a Double for division.
ratio :: Int -> Int -> Double
ratio a b = fromIntegral a / fromIntegral b
-- The confusion matrix and the metrics built from it, at one threshold.
report :: [(Bool, Double)] -> String -> Double -> IO ()
report parts suffix threshold = do
-- A part is flagged when its score reaches the threshold. Each count is
-- the parts with one combination of (flagged, defective).
let count flagged defective =
length [() | (d, s) <- parts, (s >= threshold) == flagged, d == defective]
tp = count True True
fp = count True False
fn = count False True
tn = count False False
accuracy = ratio (tp + tn) (length parts)
precision = ratio tp (tp + fp) -- of the flagged parts, how many are defective
recall = ratio tp (tp + fn) -- of the defective parts, how many are flagged
f1 = 2 * precision * recall / (precision + recall)
printf "tp_at_%s %d\n" suffix tp
printf "fp_at_%s %d\n" suffix fp
printf "fn_at_%s %d\n" suffix fn
printf "tn_at_%s %d\n" suffix tn
printf "accuracy_at_%s %.6f\n" suffix accuracy
printf "precision_at_%s %.6f\n" suffix precision
printf "recall_at_%s %.6f\n" suffix recall
printf "f1_at_%s %.6f\n" suffix f1
-- The chance that a random defective part outscores a random good one.
-- Compare every pair, scoring 2 for a win and 1 for a tie, so the count
-- stays a whole number until the final division.
auc :: [(Bool, Double)] -> Double
auc parts = ratio points (2 * length defective * length good)
where
defective = [s | (True, s) <- parts]
good = [s | (False, s) <- parts]
points = sum [pairPoints (compare d g) | d <- defective, g <- good]
pairPoints GT = 2
pairPoints EQ = 1
pairPoints LT = 0 :: Int
main :: IO ()
main = do
[path] <- getArgs
text <- readFile path
let parts = readParts text
n = length parts
defective = length (filter fst parts)
printf "parts %d\n" n
printf "defective %d\n" defective
-- Never flagging anything is right about every good part.
printf "baseline_accuracy %.6f\n" (ratio (n - defective) n)
report parts "50" 0.5
report parts "30" 0.3
printf "auc %.6f\n" (auc parts)Each part becomes a (Bool, Double) pair. One small function, count,
takes a flagged value and a defective value and counts the parts with
that combination, so the four cells of the confusion matrix are four calls
to it. Every metric is a ratio of those counts.
The AUC is a list comprehension over every defective score and every good
score, 33 times 167 pairs. compare returns GT, EQ, or LT, and
pattern matching turns those into 2, 1, and 0 points, so the total stays
an Int until the last division.
// Evaluation metrics for a binary classifier, using only the Java standard
// library.
//
// Reads each part's true label and the model's score, then reports the
// confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
// the accuracy of never flagging anything, and the area under the ROC curve.
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
class Main {
// A record is a small immutable class; Java writes its constructor and
// accessors, such as part.score(), for us.
record Part(boolean defective, double score) {}
static final Locale ROOT = Locale.ROOT; // print 0.5, never 0,5
// Sort every part into one of four counts at this threshold, then print
// the counts and the ratios built from them.
static void report(List<Part> parts, String suffix, double threshold) {
int tp = 0, fp = 0, fn = 0, tn = 0;
for (Part part : parts) {
boolean flagged = part.score() >= threshold; // flagged when the score reaches it
if (flagged && part.defective()) {
tp++;
} else if (flagged) {
fp++;
} else if (part.defective()) {
fn++;
} else {
tn++;
}
}
// Casting to double first makes the division a real one; int / int
// would throw away the fraction.
double accuracy = (double) (tp + tn) / parts.size();
double precision = (double) tp / (tp + fp); // of the flagged, how many defective
double recall = (double) tp / (tp + fn); // of the defective, how many flagged
double f1 = 2 * precision * recall / (precision + recall);
System.out.printf(ROOT, "tp_at_%s %d%n", suffix, tp);
System.out.printf(ROOT, "fp_at_%s %d%n", suffix, fp);
System.out.printf(ROOT, "fn_at_%s %d%n", suffix, fn);
System.out.printf(ROOT, "tn_at_%s %d%n", suffix, tn);
System.out.printf(ROOT, "accuracy_at_%s %.6f%n", suffix, accuracy);
System.out.printf(ROOT, "precision_at_%s %.6f%n", suffix, precision);
System.out.printf(ROOT, "recall_at_%s %.6f%n", suffix, recall);
System.out.printf(ROOT, "f1_at_%s %.6f%n", suffix, f1);
}
// The chance that a random defective part outscores a random good one.
// Compare every pair, scoring 2 for a win and 1 for a tie, so the count
// stays a whole number until the final division.
static double auc(List<Part> parts) {
List<Double> defective = new ArrayList<>(), good = new ArrayList<>();
for (Part part : parts) {
(part.defective() ? defective : good).add(part.score());
}
long points = 0;
// Unboxing to the primitive double matters: == on two Double objects
// would compare references, not values.
for (double d : defective) {
for (double g : good) {
if (d > g) {
points += 2;
} else if (d == g) {
points += 1;
}
}
}
return (double) points / (2.0 * defective.size() * good.size());
}
// Reading a file can fail with an IOException, and Java makes us say so
// with "throws".
public static void main(String[] args) throws IOException {
List<String> lines = Files.readAllLines(Path.of(args[0]));
List<Part> parts = new ArrayList<>();
int defective = 0;
for (String line : lines.subList(1, lines.size())) { // skip the header
if (line.isBlank()) {
continue;
}
String[] fields = line.split(",");
// Strings are compared with equals; == would compare objects.
Part part = new Part(fields[0].equals("1"), Double.parseDouble(fields[1]));
parts.add(part);
if (part.defective()) {
defective++;
}
}
System.out.printf(ROOT, "parts %d%n", parts.size());
System.out.printf(ROOT, "defective %d%n", defective);
// Never flagging anything is right about every good part.
System.out.printf(ROOT, "baseline_accuracy %.6f%n",
(double) (parts.size() - defective) / parts.size());
report(parts, "50", 0.5);
report(parts, "30", 0.3);
System.out.printf(ROOT, "auc %.6f%n", auc(parts));
}
}Each row becomes a Part record, a one-line immutable class whose
constructor and accessors Java writes for us. report sorts the parts into
four counts at a threshold, and every metric is a ratio of those counts,
cast to double first because int / int drops the fraction.
The AUC splits the scores into two List<Double> and compares every pair.
The loop variables are the primitive double, which matters: == on two
boxed Double objects compares references, so two parsed copies of 0.174
come out unequal and the 3 ties would be lost. A win adds 2 points and a
tie 1, in a long, until the final division.
// Evaluation metrics for a binary classifier, using only Node's built-in
// modules.
//
// Reads each part's true label and the model's score, then reports the
// confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
// the accuracy of never flagging anything, and the area under the ROC curve.
import { readFileSync } from "node:fs";
const THRESHOLDS = [["50", 0.5], ["30", 0.3]]; // name suffix, threshold
// No field here is quoted, so we split on commas and find columns by name.
function read(path) {
const [header, ...lines] = readFileSync(path, "utf8").trim().split("\n");
const columns = header.split(",");
const iLabel = columns.indexOf("defective");
const iScore = columns.indexOf("score");
return lines.map((line) => {
const fields = line.split(",");
return { defective: fields[iLabel] === "1", score: Number(fields[iScore]) };
});
}
function report(parts, suffix, threshold) {
// A part is flagged when its score reaches the threshold.
let tp = 0, fp = 0, fn = 0, tn = 0;
for (const { defective, score } of parts) {
const flagged = score >= threshold;
if (flagged && defective) tp++;
else if (flagged) fp++;
else if (defective) fn++;
else tn++;
}
const accuracy = (tp + tn) / parts.length;
const precision = tp / (tp + fp); // of the flagged parts, how many are defective
const recall = tp / (tp + fn); // of the defective parts, how many are flagged
const f1 = (2 * precision * recall) / (precision + recall);
console.log(`tp_at_${suffix} ${tp}`);
console.log(`fp_at_${suffix} ${fp}`);
console.log(`fn_at_${suffix} ${fn}`);
console.log(`tn_at_${suffix} ${tn}`);
console.log(`accuracy_at_${suffix} ${accuracy.toFixed(6)}`);
console.log(`precision_at_${suffix} ${precision.toFixed(6)}`);
console.log(`recall_at_${suffix} ${recall.toFixed(6)}`);
console.log(`f1_at_${suffix} ${f1.toFixed(6)}`);
}
// The chance that a random defective part outscores a random good one.
// Compare every pair, scoring 2 for a win and 1 for a tie, so the count
// stays a whole number until the final division.
function auc(parts) {
const defective = parts.filter((p) => p.defective).map((p) => p.score);
const good = parts.filter((p) => !p.defective).map((p) => p.score);
let points = 0;
for (const d of defective) {
for (const g of good) {
if (d > g) points += 2;
else if (d === g) points += 1;
}
}
return points / (2 * defective.length * good.length);
}
const parts = read(process.argv[2]);
const defective = parts.filter((p) => p.defective).length;
console.log(`parts ${parts.length}`);
console.log(`defective ${defective}`);
// Never flagging anything is right about every good part.
console.log(`baseline_accuracy ${((parts.length - defective) / parts.length).toFixed(6)}`);
for (const [suffix, threshold] of THRESHOLDS) report(parts, suffix, threshold);
console.log(`auc ${auc(parts).toFixed(6)}`);We split each line on commas, find the two columns by name in the header,
and turn every row into an object with a boolean defective and a numeric
score. One loop per threshold sorts every part into one of four counts:
true positives, false positives, false negatives, and true negatives. Every
metric is a ratio of those counts.
The AUC compares every defective part with every good one, 33 times 167
pairs, scoring 2 for a win and 1 for a tie. JavaScript has one number type,
a double, but Number("0.300") === 0.3 holds, so tied scores in the file
stay tied.
# Evaluation metrics for a binary classifier, using only Base Julia.
#
# Reads each part's true label and the model's score, then reports the
# confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
# the accuracy of never flagging anything, and the area under the ROC curve.
using Printf
const THRESHOLDS = (("50", 0.5), ("30", 0.3)) # name suffix, threshold
# Each part becomes a (defective, score) tuple; columns are found by name.
function read_parts(path)
lines = readlines(path)
columns = split(lines[1], ",")
i_label = findfirst(==("defective"), columns)
i_score = findfirst(==("score"), columns)
return [(fields[i_label] == "1", parse(Float64, fields[i_score]))
for fields in (split(line, ",") for line in lines[2:end])]
end
function report(parts, suffix, threshold)
# A part is flagged when its score reaches the threshold.
tp = fp = fn = tn = 0
for (defective, score) in parts
flagged = score >= threshold
if flagged && defective
tp += 1
elseif flagged
fp += 1
elseif defective
fn += 1
else
tn += 1
end
end
accuracy = (tp + tn) / length(parts) # / on two Ints gives a Float64
precision = tp / (tp + fp) # of the flagged parts, how many are defective
recall = tp / (tp + fn) # of the defective parts, how many are flagged
f1 = 2 * precision * recall / (precision + recall)
println("tp_at_$suffix $tp")
println("fp_at_$suffix $fp")
println("fn_at_$suffix $fn")
println("tn_at_$suffix $tn")
@printf("accuracy_at_%s %.6f\n", suffix, accuracy)
@printf("precision_at_%s %.6f\n", suffix, precision)
@printf("recall_at_%s %.6f\n", suffix, recall)
@printf("f1_at_%s %.6f\n", suffix, f1)
end
# The chance that a random defective part outscores a random good one.
# Compare every pair, scoring 2 for a win and 1 for a tie, so the count
# stays a whole number until the final division.
function auc(parts)
defective = [score for (is_defective, score) in parts if is_defective]
good = [score for (is_defective, score) in parts if !is_defective]
points = 0
for d in defective, g in good
if d > g
points += 2
elseif d == g
points += 1
end
end
return points / (2 * length(defective) * length(good))
end
function main(path)
parts = read_parts(path)
defective = count(first, parts)
println("parts $(length(parts))")
println("defective $defective")
# Never flagging anything is right about every good part.
@printf("baseline_accuracy %.6f\n", (length(parts) - defective) / length(parts))
for (suffix, threshold) in THRESHOLDS
report(parts, suffix, threshold)
end
@printf("auc %.6f\n", auc(parts))
end
main(ARGS[1])Each part becomes a (Bool, Float64) tuple, and one loop per threshold
sorts every part into true positives, false positives, false negatives, and
true negatives. In Julia, / on two Ints gives a Float64, so
tp / (tp + fp) needs no conversion; div is the integer division.
For the AUC, a single for d in defective, g in good loop visits all 33
times 167 pairs, scoring 2 for a win and 1 for a tie. count(first, parts)
counts the defective parts by applying first to each tuple.
-- Evaluation metrics for a binary classifier, using only Lua's standard
-- library.
--
-- Reads each part's true label and the model's score, then reports the
-- confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
-- the accuracy of never flagging anything, and the area under the ROC curve.
-- Returns a list of parts, each a table {defective = bool, score = number}.
local function read_parts(path)
local f = assert(io.open(path, "r"))
local col, i = {}, 0
for field in f:read("l"):gmatch("[^,]+") do
i = i + 1
col[field] = i -- column name -> position
end
local parts = {}
for line in f:lines() do
local fields = {}
for field in line:gmatch("[^,]+") do
fields[#fields + 1] = field
end
parts[#parts + 1] = {
defective = fields[col.defective] == "1",
score = tonumber(fields[col.score]),
}
end
f:close()
return parts
end
local function report(parts, suffix, threshold)
-- A part is flagged when its score reaches the threshold.
local tp, fp, fn, tn = 0, 0, 0, 0
for _, p in ipairs(parts) do
local flagged = p.score >= threshold
if flagged and p.defective then tp = tp + 1
elseif flagged then fp = fp + 1
elseif p.defective then fn = fn + 1
else tn = tn + 1 end
end
-- / always divides as a decimal, even between two integers.
local accuracy = (tp + tn) / #parts
local precision = tp / (tp + fp) -- of the flagged parts, how many are defective
local recall = tp / (tp + fn) -- of the defective parts, how many are flagged
local f1 = 2 * precision * recall / (precision + recall)
print("tp_at_" .. suffix .. " " .. tp)
print("fp_at_" .. suffix .. " " .. fp)
print("fn_at_" .. suffix .. " " .. fn)
print("tn_at_" .. suffix .. " " .. tn)
print(string.format("accuracy_at_%s %.6f", suffix, accuracy))
print(string.format("precision_at_%s %.6f", suffix, precision))
print(string.format("recall_at_%s %.6f", suffix, recall))
print(string.format("f1_at_%s %.6f", suffix, f1))
end
-- The chance that a random defective part outscores a random good one.
-- Compare every pair, scoring 2 for a win and 1 for a tie, so the count
-- stays a whole number until the final division.
local function auc(parts)
local defective, good = {}, {}
for _, p in ipairs(parts) do
local list = p.defective and defective or good
list[#list + 1] = p.score
end
local points = 0
for _, d in ipairs(defective) do
for _, g in ipairs(good) do
if d > g then points = points + 2
elseif d == g then points = points + 1 end
end
end
return points / (2 * #defective * #good)
end
local parts = read_parts(arg[1])
local defective = 0
for _, p in ipairs(parts) do
if p.defective then defective = defective + 1 end
end
print("parts " .. #parts)
print("defective " .. defective)
-- Never flagging anything is right about every good part.
print(string.format("baseline_accuracy %.6f", (#parts - defective) / #parts))
report(parts, "50", 0.5)
report(parts, "30", 0.3)
print(string.format("auc %.6f", auc(parts)))Each part becomes a small table with a defective flag and a score, and
one loop per threshold sorts every part into one of four counts: true
positives, false positives, false negatives, and true negatives. Every
metric is a ratio of those counts.
tonumber("0.174") gives a float, and equal strings in the file give
equal floats, so the three tied pairs compare as ties. The AUC loop scores
2 for a win and 1 for a tie across all 33 times 167 pairs. Lua’s /
always returns a float, so no division needs a cast. LuaRocks isn’t
installed.
(* Evaluation metrics for a binary classifier, using only OCaml's standard
library.
Reads each part's true label and the model's score, then reports the
confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
the accuracy of never flagging anything, and the area under the ROC
curve. *)
(* Every part as (defective, score), with each column found by its header
name. *)
let read_parts path =
In_channel.with_open_text path (fun ic ->
let lines = In_channel.input_lines ic in
let header = String.split_on_char ',' (List.hd lines) in
let index name =
match List.find_index (String.equal name) header with
| Some i -> i
| None -> failwith ("column not found: " ^ name)
in
List.filter_map
(fun line ->
if line = "" then None
else
let fields = String.split_on_char ',' line in
let field name = List.nth fields (index name) in
Some (field "defective" = "1", float_of_string (field "score")))
(List.tl lines))
(* OCaml never mixes int and float: float_of_int converts, and /. is float
division. *)
let ratio a b = float_of_int a /. float_of_int b
let report parts suffix threshold =
(* A part is flagged when its score reaches the threshold. *)
let tp = ref 0 and fp = ref 0 and fn = ref 0 and tn = ref 0 in
List.iter
(fun (defective, score) ->
let counter =
match (score >= threshold, defective) with
| true, true -> tp
| true, false -> fp
| false, true -> fn
| false, false -> tn
in
incr counter)
parts;
let tp = !tp and fp = !fp and fn = !fn and tn = !tn in
let accuracy = ratio (tp + tn) (List.length parts) in
(* Of the flagged parts, how many are defective. *)
let precision = ratio tp (tp + fp) in
(* Of the defective parts, how many are flagged. *)
let recall = ratio tp (tp + fn) in
let f1 = 2. *. precision *. recall /. (precision +. recall) in
Printf.printf "tp_at_%s %d\n" suffix tp;
Printf.printf "fp_at_%s %d\n" suffix fp;
Printf.printf "fn_at_%s %d\n" suffix fn;
Printf.printf "tn_at_%s %d\n" suffix tn;
Printf.printf "accuracy_at_%s %.6f\n" suffix accuracy;
Printf.printf "precision_at_%s %.6f\n" suffix precision;
Printf.printf "recall_at_%s %.6f\n" suffix recall;
Printf.printf "f1_at_%s %.6f\n" suffix f1
(* The chance that a random defective part outscores a random good one.
Compare every pair, scoring 2 for a win and 1 for a tie, so the count
stays a whole number until the final division. *)
let auc parts =
let scores wanted =
List.filter_map (fun (d, s) -> if d = wanted then Some s else None) parts
in
let defective = scores true and good = scores false in
let points = ref 0 in
List.iter
(fun d ->
List.iter
(fun g ->
if d > g then points := !points + 2
else if d = g then incr points)
good)
defective;
ratio !points (2 * List.length defective * List.length good)
let () =
let parts = read_parts Sys.argv.(1) in
let n = List.length parts in
let defective = List.length (List.filter fst parts) in
Printf.printf "parts %d\n" n;
Printf.printf "defective %d\n" defective;
(* Never flagging anything is right about every good part. *)
Printf.printf "baseline_accuracy %.6f\n" (ratio (n - defective) n);
report parts "50" 0.5;
report parts "30" 0.3;
Printf.printf "auc %.6f\n" (auc parts)Each part becomes a (bool, float) pair. One pass per threshold matches
on (flagged, defective) to pick which of four ref counters to bump:
true positives, false positives, false negatives, or true negatives.
Every metric is a ratio of those counts.
OCaml never mixes int and float, so a small ratio helper converts
both counts with float_of_int and divides with /.. The AUC walks every
defective score against every good one, 33 times 167 pairs, adding 2 for a
win and 1 for a tie, so the total stays a whole number until the end.
% Evaluation metrics for a binary classifier, using only core GNU Octave.
%
% Reads each part's true label and the model's score, then reports the
% confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
% the accuracy of never flagging anything, and the area under the ROC curve.
% The named column of a CSV file, as a column cell array of strings. Wrap it
% in str2double to get numbers.
function values = read_column(path, name)
lines = strsplit(strtrim(fileread(path)), "\n");
header = strsplit(strtrim(lines{1}), ",");
index = find(strcmp(header, name));
if isempty(index)
error("no column %s", name);
end
values = cell(numel(lines) - 1, 1);
for row = 2:numel(lines)
fields = strsplit(strtrim(lines{row}), ",");
values{row - 1} = fields{index};
end
end
% Counts and metrics at one threshold. Every comparison below works on the
% whole column at once and gives a logical vector, and sum counts its trues.
function report(defective, score, suffix, threshold)
flagged = score >= threshold; % a part is flagged when it reaches the threshold
tp = sum(flagged & defective);
fp = sum(flagged & !defective);
fn = sum(!flagged & defective);
tn = sum(!flagged & !defective);
accuracy = (tp + tn) / numel(score);
precision = tp / (tp + fp); % of the flagged parts, how many are defective
recall = tp / (tp + fn); % of the defective parts, how many are flagged
f1 = 2 * precision * recall / (precision + recall);
printf("tp_at_%s %d\n", suffix, tp);
printf("fp_at_%s %d\n", suffix, fp);
printf("fn_at_%s %d\n", suffix, fn);
printf("tn_at_%s %d\n", suffix, tn);
printf("accuracy_at_%s %.6f\n", suffix, accuracy);
printf("precision_at_%s %.6f\n", suffix, precision);
printf("recall_at_%s %.6f\n", suffix, recall);
printf("f1_at_%s %.6f\n", suffix, f1);
end
args = argv();
defective = strcmp(read_column(args{1}, "defective"), "1"); % logical column
score = str2double(read_column(args{1}, "score"));
n = numel(score);
positives = sum(defective);
printf("parts %d\n", n);
printf("defective %d\n", positives);
% Never flagging anything is right about every good part.
printf("baseline_accuracy %.6f\n", (n - positives) / n);
report(defective, score, "50", 0.5);
report(defective, score, "30", 0.3);
% AUC: the chance that a random defective part outscores a random good one.
% Comparing a column with a row broadcasts: d > g' is a matrix with one row
% per defective part and one column per good part, holding every pairwise
% comparison, and (:) flattens it so sum counts them all. A win scores 2 and
% a tie 1, so the count stays a whole number until the final division.
d = score(defective);
g = score(!defective);
points = 2 * sum((d > g')(:)) + sum((d == g')(:));
printf("auc %.6f\n", points / (2 * numel(d) * numel(g)));No loop sorts the parts into counts. score >= threshold compares the whole
column at once and gives a logical vector, and & and ! combine it with
the labels, so each of the four counts is one sum.
For the AUC, d > g' compares a column of 33 defective scores with a row
of 167 good ones. Octave broadcasts the two into a 33-by-167 matrix of
every pairwise result, and (:) flattens it for sum. Ties count half
through the matching == matrix. Octave Forge packages, such as
statistics, aren’t installed.
{ Evaluation metrics for a binary classifier, using only Free Pascal's
run-time library.
Reads each part's true label and the model's score, then reports the
confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
the accuracy of never flagging anything, and the area under the ROC
curve. }
program EvaluationMetrics;
{$mode objfpc}{$H+} // 32-bit Integer, and strings longer than 255 characters
uses
SysUtils;
type
TPart = record
Defective: Boolean;
Score: Double;
end;
TPartArray = array of TPart;
var
Parts: TPartArray;
Part: TPart;
Defective: Integer;
// Read every part's label and score.
function ReadParts(const Path: String): TPartArray;
var
F: TextFile;
Line: String;
Fields: TStringArray;
DefectiveCol, ScoreCol, I: Integer;
P: TPart;
begin
Result := nil;
AssignFile(F, Path);
Reset(F);
ReadLn(F, Line);
Fields := Line.Split(',');
DefectiveCol := -1;
ScoreCol := -1;
for I := 0 to High(Fields) do
if Fields[I] = 'defective' then DefectiveCol := I
else if Fields[I] = 'score' then ScoreCol := I;
if (DefectiveCol < 0) or (ScoreCol < 0) then
raise Exception.Create('column not found');
while not Eof(F) do
begin
ReadLn(F, Line);
Fields := Line.Split(',');
P.Defective := Fields[DefectiveCol] = '1';
// StrToFloat reads "0.174"; equal strings give equal Doubles.
P.Score := StrToFloat(Fields[ScoreCol]);
// Insert(Value, Array, Length(Array)) appends.
Insert(P, Result, Length(Result));
end;
CloseFile(F);
end;
// Threshold is a Double, so the literal it's given is rounded the same way
// as the scores before they're compared.
procedure Report(const Suffix: String; Threshold: Double);
var
TP, FP, FN, TN: Integer;
Flagged: Boolean;
Accuracy, Precision, Recall, F1: Double;
begin
TP := 0; FP := 0; FN := 0; TN := 0;
for Part in Parts do
begin
// A part is flagged when its score reaches the threshold.
Flagged := Part.Score >= Threshold;
if Flagged and Part.Defective then Inc(TP)
else if Flagged then Inc(FP)
else if Part.Defective then Inc(FN)
else Inc(TN);
end;
// / always divides as a decimal; div is whole-number division.
Accuracy := (TP + TN) / Length(Parts);
Precision := TP / (TP + FP); // of the flagged parts, how many are defective
Recall := TP / (TP + FN); // of the defective parts, how many are flagged
F1 := 2 * Precision * Recall / (Precision + Recall);
WriteLn('tp_at_', Suffix, ' ', TP);
WriteLn('fp_at_', Suffix, ' ', FP);
WriteLn('fn_at_', Suffix, ' ', FN);
WriteLn('tn_at_', Suffix, ' ', TN);
WriteLn('accuracy_at_', Suffix, ' ', Accuracy:0:6);
WriteLn('precision_at_', Suffix, ' ', Precision:0:6);
WriteLn('recall_at_', Suffix, ' ', Recall:0:6);
WriteLn('f1_at_', Suffix, ' ', F1:0:6);
end;
// The chance that a random defective part outscores a random good one.
// Compare every pair, scoring 2 for a win and 1 for a tie, so the count
// stays a whole number until the final division.
function AUC: Double;
var
D, G: TPart;
Points: Integer;
begin
Points := 0;
for D in Parts do
if D.Defective then
for G in Parts do
if not G.Defective then
if D.Score > G.Score then Inc(Points, 2)
else if D.Score = G.Score then Inc(Points);
Result := Points / (2 * Defective * (Length(Parts) - Defective));
end;
begin
Parts := ReadParts(ParamStr(1));
Defective := 0;
for Part in Parts do
if Part.Defective then Inc(Defective);
WriteLn('parts ', Length(Parts));
WriteLn('defective ', Defective);
// Never flagging anything is right about every good part.
WriteLn('baseline_accuracy ', (Length(Parts) - Defective) / Length(Parts):0:6);
Report('50', 0.5);
Report('30', 0.3);
WriteLn('auc ', AUC:0:6);
end.Each part is a record with a Boolean label and a Double score, held in
a dynamic array. One loop per threshold sorts every part into one of four
counts: true positives, false positives, false negatives, and true
negatives. Every metric is a ratio of those counts.
The AUC uses two nested for ... in loops over the parts, skipping the
wrong labels, and scores 2 for a win and 1 for a tie in an Integer. The
threshold is a Double parameter, so it’s rounded the same way as the
scores it’s compared with. / gives a decimal even between integers.
# Evaluation metrics for a binary classifier, using only Perl and its core
# modules.
#
# Reads each part's true label and the model's score, then reports the
# confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
# the accuracy of never flagging anything, and the area under the ROC curve.
use strict;
use warnings;
# Returns a list of parts, each an array reference [defective, score].
sub read_parts {
my ($path) = @_;
open my $fh, '<', $path or die "can't open $path: $!";
chomp(my $header = <$fh>);
my @fields = split /,/, $header;
my %col = map { $fields[$_] => $_ } 0 .. $#fields; # name -> position
my @parts;
while (my $line = <$fh>) {
chomp $line;
my @row = split /,/, $line;
# == compares as numbers, so "1" and "1.0" would both count.
push @parts, [ $row[ $col{defective} ] == 1, $row[ $col{score} ] ];
}
close $fh;
return @parts;
}
sub report {
my ($parts, $suffix, $threshold) = @_;
# A part is flagged when its score reaches the threshold.
my ($tp, $fp, $fn, $tn) = (0, 0, 0, 0);
for my $part (@$parts) {
my ($defective, $score) = @$part;
my $flagged = $score >= $threshold;
if ($flagged && $defective) { $tp++ }
elsif ($flagged) { $fp++ }
elsif ($defective) { $fn++ }
else { $tn++ }
}
my $accuracy = ($tp + $tn) / @$parts;
my $precision = $tp / ($tp + $fp); # of the flagged parts, how many are defective
my $recall = $tp / ($tp + $fn); # of the defective parts, how many are flagged
my $f1 = 2 * $precision * $recall / ($precision + $recall);
print "tp_at_$suffix $tp\n";
print "fp_at_$suffix $fp\n";
print "fn_at_$suffix $fn\n";
print "tn_at_$suffix $tn\n";
printf "accuracy_at_%s %.6f\n", $suffix, $accuracy;
printf "precision_at_%s %.6f\n", $suffix, $precision;
printf "recall_at_%s %.6f\n", $suffix, $recall;
printf "f1_at_%s %.6f\n", $suffix, $f1;
}
# The chance that a random defective part outscores a random good one.
# Compare every pair, scoring 2 for a win and 1 for a tie, so the count
# stays a whole number until the final division. <=> returns 1, 0, or -1,
# so adding 1 gives 2 for a win, 1 for a tie, and 0 for a loss.
sub auc {
my @parts = @_;
my @defective = map { $_->[1] } grep { $_->[0] } @parts;
my @good = map { $_->[1] } grep { !$_->[0] } @parts;
my $points = 0;
for my $d (@defective) {
$points += ($d <=> $_) + 1 for @good;
}
return $points / (2 * @defective * @good);
}
my @parts = read_parts($ARGV[0]);
my $defective = grep { $_->[0] } @parts; # grep in scalar context counts
print 'parts ', scalar @parts, "\n";
print "defective $defective\n";
# Never flagging anything is right about every good part.
printf "baseline_accuracy %.6f\n", (@parts - $defective) / @parts;
report(\@parts, '50', 0.5);
report(\@parts, '30', 0.3);
printf "auc %.6f\n", auc(@parts);Each part becomes a two-element array reference, a label and a score, and one loop per threshold sorts every part into one of four counts: true positives, false positives, false negatives, and true negatives. Every metric is a ratio of those counts.
For the AUC we lean on the spaceship operator: $d <=> $g returns 1, 0,
or -1, so adding 1 scores a win as 2, a tie as 1, and a loss as 0, with no
branching. grep in scalar context counts the defective parts. CPAN
modules, such as Statistics::ROC, aren’t installed.
<?php
// Evaluation metrics for a binary classifier, using only PHP and the
// extensions built into it.
//
// Reads each part's true label and the model's score, then reports the
// confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
// the accuracy of never flagging anything, and the area under the ROC curve.
declare(strict_types=1);
// Returns a list of parts, each [bool defective, float score].
function read_parts(string $path): array
{
$lines = file($path, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$col = array_flip(explode(',', array_shift($lines))); // name => position
$parts = [];
foreach ($lines as $line) {
$row = explode(',', $line);
$parts[] = [$row[$col['defective']] === '1', (float) $row[$col['score']]];
}
return $parts;
}
function report(array $parts, string $suffix, float $threshold): void
{
// A part is flagged when its score reaches the threshold.
$tp = $fp = $fn = $tn = 0;
foreach ($parts as [$defective, $score]) {
$flagged = $score >= $threshold;
if ($flagged && $defective) {
$tp++;
} elseif ($flagged) {
$fp++;
} elseif ($defective) {
$fn++;
} else {
$tn++;
}
}
$accuracy = ($tp + $tn) / count($parts);
$precision = $tp / ($tp + $fp); // of the flagged parts, how many are defective
$recall = $tp / ($tp + $fn); // of the defective parts, how many are flagged
$f1 = 2 * $precision * $recall / ($precision + $recall);
echo "tp_at_$suffix $tp\n";
echo "fp_at_$suffix $fp\n";
echo "fn_at_$suffix $fn\n";
echo "tn_at_$suffix $tn\n";
printf("accuracy_at_%s %.6f\n", $suffix, $accuracy);
printf("precision_at_%s %.6f\n", $suffix, $precision);
printf("recall_at_%s %.6f\n", $suffix, $recall);
printf("f1_at_%s %.6f\n", $suffix, $f1);
}
// The chance that a random defective part outscores a random good one.
// Compare every pair, scoring 2 for a win and 1 for a tie, so the count
// stays a whole number until the final division. The spaceship operator
// <=> returns 1, 0, or -1, so adding 1 gives 2, 1, or 0 points.
function auc(array $parts): float
{
$defective = [];
$good = [];
foreach ($parts as [$is_defective, $score]) {
if ($is_defective) {
$defective[] = $score;
} else {
$good[] = $score;
}
}
$points = 0;
foreach ($defective as $d) {
foreach ($good as $g) {
$points += ($d <=> $g) + 1;
}
}
return $points / (2 * count($defective) * count($good));
}
$parts = read_parts($argv[1]);
$defective = count(array_filter($parts, fn($part) => $part[0]));
echo 'parts ', count($parts), "\n";
echo "defective $defective\n";
// Never flagging anything is right about every good part.
printf("baseline_accuracy %.6f\n", (count($parts) - $defective) / count($parts));
report($parts, '50', 0.5);
report($parts, '30', 0.3);
printf("auc %.6f\n", auc($parts));Each part becomes a pair of a bool and a float, and one loop per
threshold sorts every part into one of four counts: true positives, false
positives, false negatives, and true negatives. foreach unpacks each
pair straight into two variables. Every metric is a ratio of those counts.
For the AUC we use the spaceship operator: $d <=> $g returns 1, 0, or
-1, so adding 1 scores a win as 2, a tie as 1, and a loss as 0 across all
33 times 167 pairs. array_filter counts the defective parts. Composer
packages aren’t installed.
-- Evaluation metrics for a binary classifier, in PostgreSQL.
--
-- Reads each part's true label and the model's score, then reports the
-- confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
-- the accuracy of never flagging anything, and the area under the ROC curve.
WITH parts AS (
-- run.sh loads both columns as double precision. The label is 0 or 1,
-- so we turn it into a boolean; scores that are equal text in the file
-- load as equal doubles, so ties stay ties.
SELECT defective = 1 AS defective, score FROM data
),
thresholds (suffix, threshold) AS (
VALUES ('50', 0.5::double precision), ('30', 0.3)
),
counts AS (
-- One row per threshold. A part is flagged when its score reaches the
-- threshold; FILTER counts only the rows its condition keeps.
SELECT suffix, threshold,
count(*) FILTER (WHERE score >= threshold AND defective) AS tp,
count(*) FILTER (WHERE score >= threshold AND NOT defective) AS fp,
count(*) FILTER (WHERE score < threshold AND defective) AS fn,
count(*) FILTER (WHERE score < threshold AND NOT defective) AS tn
FROM thresholds CROSS JOIN parts
GROUP BY suffix, threshold
),
metrics AS (
-- count(*) gives whole numbers, and dividing integers in SQL drops the
-- fraction, so we cast to numeric first.
SELECT suffix, threshold, tp, fp, fn, tn,
(tp + tn)::numeric / (tp + fp + fn + tn) AS accuracy,
tp::numeric / (tp + fp) AS precision, -- of the flagged, how many defective
tp::numeric / (tp + fn) AS recall -- of the defective, how many flagged
FROM counts
),
auc AS (
-- The chance that a random defective part outscores a random good one.
-- Join every defective part with every good one, score 2 for a win and
-- 1 for a tie, and divide the whole-number total once at the end.
SELECT sum(CASE WHEN d.score > g.score THEN 2
WHEN d.score = g.score THEN 1
ELSE 0 END)::numeric / (2 * count(*)) AS auc
FROM parts AS d JOIN parts AS g ON d.defective AND NOT g.defective
),
summary AS (
SELECT count(*) AS parts, count(*) FILTER (WHERE defective) AS defective
FROM parts
)
SELECT name, value
FROM (
-- Rounding a numeric to 6 places prints 6 decimals. Rows sort by
-- section, then threshold from high to low, then line within a
-- threshold's group of eight; rows outside the groups use threshold 0.
SELECT 1 AS section, 0 AS threshold, 0 AS line, 'parts' AS name, parts::text AS value
FROM summary
UNION ALL SELECT 2, 0, 0, 'defective', defective::text FROM summary
UNION ALL SELECT 3, 0, 0, 'baseline_accuracy',
round((parts - defective)::numeric / parts, 6)::text FROM summary
UNION ALL SELECT 4, threshold, 1, 'tp_at_' || suffix, tp::text FROM metrics
UNION ALL SELECT 4, threshold, 2, 'fp_at_' || suffix, fp::text FROM metrics
UNION ALL SELECT 4, threshold, 3, 'fn_at_' || suffix, fn::text FROM metrics
UNION ALL SELECT 4, threshold, 4, 'tn_at_' || suffix, tn::text FROM metrics
UNION ALL SELECT 4, threshold, 5, 'accuracy_at_' || suffix, round(accuracy, 6)::text
FROM metrics
UNION ALL SELECT 4, threshold, 6, 'precision_at_' || suffix, round(precision, 6)::text
FROM metrics
UNION ALL SELECT 4, threshold, 7, 'recall_at_' || suffix, round(recall, 6)::text
FROM metrics
UNION ALL SELECT 4, threshold, 8, 'f1_at_' || suffix,
round(2 * precision * recall / (precision + recall), 6)::text
FROM metrics
UNION ALL SELECT 5, 0, 0, 'auc', round(auc, 6)::text FROM auc
) AS results
ORDER BY section, threshold DESC, line;We never loop. A two-row VALUES list holds the thresholds, and a cross
join pairs each threshold with all 200 parts. GROUP BY then gives one row
per threshold, and count(*) FILTER (WHERE ...) fills in each cell of the
confusion matrix.
The AUC is a self-join: every defective part joined to every good one, 5,511
rows. A CASE scores each pair 2, 1, or 0, and sum adds them up. Counts
are integers, and integer division drops the fraction (7 / 2 is 3), so we
cast to numeric before dividing. Sort columns keep the output in
order, since a query’s rows have none of their own.
% Evaluation metrics for a binary classifier, using SWI-Prolog and the
% libraries that come with it.
%
% Reads each part's true label and the model's score, then reports the
% confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
% the accuracy of never flagging anything, and the area under the ROC curve.
:- initialization(main, main).
% read_parts(+Path, -Parts): every part as Defective-Score, in file order,
% with each column found by its header name. Defective is 1 or 0.
read_parts(Path, Parts) :-
read_file_to_string(Path, Text, []),
split_string(Text, "\n", "", [HeaderLine|Lines]),
split_string(HeaderLine, ",", "", Header),
nth0(D, Header, "defective"),
nth0(S, Header, "score"),
findall(Defective-Score,
( member(Line, Lines),
Line \== "",
split_string(Line, ",", "", Fields),
nth0(D, Fields, DefectiveText),
nth0(S, Fields, ScoreText),
number_string(Defective, DefectiveText),
number_string(Score, ScoreText) ),
Parts).
% outcome(+Threshold, +Part, -Outcome): which cell of the confusion matrix
% a part lands in. A part is flagged when its score reaches the threshold.
outcome(Threshold, 1-Score, tp) :- Score >= Threshold.
outcome(Threshold, 0-Score, fp) :- Score >= Threshold.
outcome(Threshold, 1-Score, fn) :- Score < Threshold.
outcome(Threshold, 0-Score, tn) :- Score < Threshold.
% report(+Parts, +Suffix, +Threshold): print the confusion matrix and the
% metrics built from it.
report(Parts, Suffix, Threshold) :-
% aggregate_all(count, Goal, Count) counts the ways Goal succeeds.
aggregate_all(count, (member(P, Parts), outcome(Threshold, P, tp)), TP),
aggregate_all(count, (member(P, Parts), outcome(Threshold, P, fp)), FP),
aggregate_all(count, (member(P, Parts), outcome(Threshold, P, fn)), FN),
aggregate_all(count, (member(P, Parts), outcome(Threshold, P, tn)), TN),
length(Parts, N),
Accuracy is (TP + TN) / N,
Precision is TP / (TP + FP), % of the flagged parts, how many are defective
Recall is TP / (TP + FN), % of the defective parts, how many are flagged
F1 is 2 * Precision * Recall / (Precision + Recall),
format("tp_at_~w ~d~n", [Suffix, TP]),
format("fp_at_~w ~d~n", [Suffix, FP]),
format("fn_at_~w ~d~n", [Suffix, FN]),
format("tn_at_~w ~d~n", [Suffix, TN]),
format("accuracy_at_~w ~6f~n", [Suffix, Accuracy]),
format("precision_at_~w ~6f~n", [Suffix, Precision]),
format("recall_at_~w ~6f~n", [Suffix, Recall]),
format("f1_at_~w ~6f~n", [Suffix, F1]).
% pair_points(+Defective, +Good, -Points): 2 when the defective part
% scores higher, 1 for an exact tie, and 0 otherwise.
pair_points(D, G, 2) :- D > G.
pair_points(D, G, 1) :- D =:= G.
pair_points(D, G, 0) :- D < G.
% auc(+Parts, -AUC): the chance that a random defective part outscores a
% random good one. Every pair scores 2 for a win and 1 for a tie, so the
% count stays a whole number until the final division.
auc(Parts, AUC) :-
findall(S, member(1-S, Parts), Defective),
findall(S, member(0-S, Parts), Good),
aggregate_all(sum(Points),
( member(D, Defective), member(G, Good),
pair_points(D, G, Points) ),
Total),
length(Defective, P),
length(Good, N),
AUC is Total / (2 * P * N).
main :-
current_prolog_flag(argv, [Path|_]),
read_parts(Path, Parts),
length(Parts, N),
aggregate_all(count, member(1-_, Parts), Defective),
format("parts ~d~n", [N]),
format("defective ~d~n", [Defective]),
% Never flagging anything is right about every good part.
% / of two integers gives a float, or an integer when it divides
% exactly; ~6f prints either with six decimals.
Baseline is (N - Defective) / N,
format("baseline_accuracy ~6f~n", [Baseline]),
report(Parts, "50", 0.5),
report(Parts, "30", 0.3),
auc(Parts, AUC),
format("auc ~6f~n", [AUC]).Each part becomes a term Defective-Score. Four clauses of outcome/3
say which cell of the confusion matrix a part lands in, and
aggregate_all(count, ...) counts the parts in each cell at a threshold.
Every metric is a ratio of those counts.
For the AUC, three clauses of pair_points/3 give 2 for a win, 1 for a
tie (=:= compares numbers by value), and 0 for a loss, and
aggregate_all with sum(Points) adds them over every defective and good
pair, 33 times 167 of them. The total stays an integer until the last
division.
# Evaluation metrics for a binary classifier, using only base R.
#
# Reads each part's true label and the model's score, then reports the
# confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
# the accuracy of never flagging anything, and the area under the ROC curve.
args <- commandArgs(trailingOnly = TRUE)
data <- read.csv(args[1])
defective <- data$defective == 1
score <- data$score
n <- length(score)
report <- function(suffix, threshold) {
# A part is flagged when its score reaches the threshold. Each count is
# the sum of a logical vector, since TRUE counts as 1.
flagged <- score >= threshold
tp <- sum(flagged & defective)
fp <- sum(flagged & !defective)
fn <- sum(!flagged & defective)
tn <- sum(!flagged & !defective)
accuracy <- (tp + tn) / n
precision <- tp / (tp + fp) # of the flagged parts, how many are defective
recall <- tp / (tp + fn) # of the defective parts, how many are flagged
f1 <- 2 * precision * recall / (precision + recall)
cat(sprintf("tp_at_%s %d\n", suffix, tp))
cat(sprintf("fp_at_%s %d\n", suffix, fp))
cat(sprintf("fn_at_%s %d\n", suffix, fn))
cat(sprintf("tn_at_%s %d\n", suffix, tn))
cat(sprintf("accuracy_at_%s %.6f\n", suffix, accuracy))
cat(sprintf("precision_at_%s %.6f\n", suffix, precision))
cat(sprintf("recall_at_%s %.6f\n", suffix, recall))
cat(sprintf("f1_at_%s %.6f\n", suffix, f1))
}
# The chance that a random defective part outscores a random good one.
# outer() builds the full table of defective-versus-good comparisons, one
# row per defective part. A win scores 2 and a tie 1, so the count stays a
# whole number until the final division.
auc <- function() {
bad <- score[defective]
good <- score[!defective]
points <- sum(2 * outer(bad, good, ">") + outer(bad, good, "=="))
points / (2 * length(bad) * length(good))
}
cat(sprintf("parts %d\n", n))
cat(sprintf("defective %d\n", sum(defective)))
# Never flagging anything is right about every good part.
cat(sprintf("baseline_accuracy %.6f\n", (n - sum(defective)) / n))
report("50", 0.5)
report("30", 0.3)
cat(sprintf("auc %.6f\n", auc()))read.csv() reads both columns as numbers, and the code never loops over
parts. score >= threshold is a logical vector, and each count is the
sum() of two such vectors combined with &, since TRUE counts as 1.
For the AUC, outer(bad, good, ">") builds a 33 by 167 logical matrix
with one cell for every pair of a defective part and a good one. Twice its
sum plus the sum of the "==" matrix gives the points. The pROC package
has an auc() function, but it isn’t installed.
# Evaluation metrics for a binary classifier, using only the standard
# library.
#
# Reads each part's true label and the model's score, then reports the
# confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
# the accuracy of never flagging anything, and the area under the ROC curve.
THRESHOLDS = [["50", 0.5], ["30", 0.3]].freeze # name suffix, threshold
# The csv library is a bundled gem since Ruby 3.4, so split lines by hand
# (no quoted fields here). Each part becomes [defective?, score].
def read(path)
header, *lines = File.readlines(path, chomp: true)
columns = header.split(",")
i_label = columns.index("defective")
i_score = columns.index("score")
lines.map do |line|
fields = line.split(",")
[fields[i_label] == "1", Float(fields[i_score])]
end
end
def report(parts, suffix, threshold)
# A part is flagged when its score reaches the threshold.
tp = fp = fn = tn = 0
parts.each do |defective, score|
flagged = score >= threshold
if flagged && defective then tp += 1
elsif flagged then fp += 1
elsif defective then fn += 1
else tn += 1
end
end
accuracy = (tp + tn).fdiv(parts.length)
precision = tp.fdiv(tp + fp) # of the flagged parts, how many are defective
recall = tp.fdiv(tp + fn) # of the defective parts, how many are flagged
f1 = 2 * precision * recall / (precision + recall)
puts "tp_at_#{suffix} #{tp}"
puts "fp_at_#{suffix} #{fp}"
puts "fn_at_#{suffix} #{fn}"
puts "tn_at_#{suffix} #{tn}"
puts format("accuracy_at_%s %.6f", suffix, accuracy)
puts format("precision_at_%s %.6f", suffix, precision)
puts format("recall_at_%s %.6f", suffix, recall)
puts format("f1_at_%s %.6f", suffix, f1)
end
# The chance that a random defective part outscores a random good one.
# Compare every pair, scoring 2 for a win and 1 for a tie, so the count
# stays a whole number until the final division.
def auc(parts)
defective, good = parts.partition(&:first).map { |group| group.map(&:last) }
points = 0
defective.each do |d|
good.each do |g|
points += 2 if d > g
points += 1 if d == g
end
end
points.fdiv(2 * defective.length * good.length)
end
parts = read(ARGV[0])
defective = parts.count(&:first)
puts "parts #{parts.length}"
puts "defective #{defective}"
# Never flagging anything is right about every good part.
puts format("baseline_accuracy %.6f", (parts.length - defective).fdiv(parts.length))
THRESHOLDS.each { |suffix, threshold| report(parts, suffix, threshold) }
puts format("auc %.6f", auc(parts))We split each line on commas and turn every row into a pair of a boolean
and a Float. Float() raises an error on text that isn’t a number, where
to_f would quietly return 0.0. One loop per threshold sorts every part
into true positives, false positives, false negatives, and true negatives.
For the AUC, partition(&:first) splits the parts into defective and good
in one call, and two nested loops compare all 33 times 167 pairs, scoring 2
for a win and 1 for a tie. In Ruby, 3 / 4 is 0, so every ratio uses
fdiv.
//! Evaluation metrics for a binary classifier, using only the standard
//! library.
//!
//! Reads each part's true label and the model's score, then reports the
//! confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
//! the accuracy of never flagging anything, and the area under the ROC curve.
use std::env;
use std::fs;
const THRESHOLDS: [(&str, f64); 2] = [("50", 0.5), ("30", 0.3)]; // name suffix, threshold
/// Each part becomes (defective, score). The file has no quoted fields, so
/// commas suffice; columns are found by name in the header.
fn read(path: &str) -> Vec<(bool, f64)> {
let text = fs::read_to_string(path).expect("cannot read data file");
let mut lines = text.lines();
let columns: Vec<&str> = lines.next().expect("empty file").split(',').collect();
let column = |name| columns.iter().position(|&c| c == name).expect("missing column");
let (i_label, i_score) = (column("defective"), column("score"));
lines
.map(|line| {
let fields: Vec<&str> = line.split(',').collect();
let score: f64 = fields[i_score].parse().expect("score must be a number");
(fields[i_label] == "1", score)
})
.collect()
}
fn report(parts: &[(bool, f64)], suffix: &str, threshold: f64) {
// A part is flagged when its score reaches the threshold.
let (mut tp, mut fp, mut fn_, mut tn) = (0u32, 0u32, 0u32, 0u32); // `fn` is a keyword
for &(defective, score) in parts {
match (score >= threshold, defective) {
(true, true) => tp += 1,
(true, false) => fp += 1,
(false, true) => fn_ += 1,
(false, false) => tn += 1,
}
}
let ratio = |a: u32, b: u32| a as f64 / b as f64;
let accuracy = ratio(tp + tn, parts.len() as u32);
let precision = ratio(tp, tp + fp); // of the flagged parts, how many are defective
let recall = ratio(tp, tp + fn_); // of the defective parts, how many are flagged
let f1 = 2.0 * precision * recall / (precision + recall);
println!("tp_at_{suffix} {tp}");
println!("fp_at_{suffix} {fp}");
println!("fn_at_{suffix} {fn_}");
println!("tn_at_{suffix} {tn}");
println!("accuracy_at_{suffix} {accuracy:.6}");
println!("precision_at_{suffix} {precision:.6}");
println!("recall_at_{suffix} {recall:.6}");
println!("f1_at_{suffix} {f1:.6}");
}
/// The chance that a random defective part outscores a random good one.
/// Compare every pair, scoring 2 for a win and 1 for a tie, so the count
/// stays a whole number until the final division.
fn auc(parts: &[(bool, f64)]) -> f64 {
let scores = |want: bool| -> Vec<f64> {
parts.iter().filter(|p| p.0 == want).map(|p| p.1).collect()
};
let (defective, good) = (scores(true), scores(false));
let mut points = 0u64;
for &d in &defective {
for &g in &good {
if d > g {
points += 2;
} else if d == g {
points += 1;
}
}
}
points as f64 / (2 * defective.len() * good.len()) as f64
}
fn main() {
let path = env::args().nth(1).expect("usage: main <data.csv>");
let parts = read(&path);
let n = parts.len();
let defective = parts.iter().filter(|p| p.0).count();
println!("parts {n}");
println!("defective {defective}");
// Never flagging anything is right about every good part.
println!("baseline_accuracy {:.6}", (n - defective) as f64 / n as f64);
for (suffix, threshold) in THRESHOLDS {
report(&parts, suffix, threshold);
}
println!("auc {:.6}", auc(&parts));
}Each part becomes a (bool, f64) tuple. One loop per threshold matches on
the pair (flagged, defective), and the compiler checks that all four
cases are covered. The false-negative count is fn_, because fn is a
keyword and let fn = 1 doesn’t compile.
The counts are u32, so a small closure, ratio, converts both sides
with as f64 before dividing. The AUC loops over all 33 times 167 pairs,
scoring 2 for a win and 1 for a tie in a u64. No crates are installed in
this image, and the run uses rustc directly.
// Evaluation metrics for a binary classifier, using only Swift's standard
// library and Foundation, which ships with it.
//
// Reads each part's true label and the model's score, then reports the
// confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
// the accuracy of never flagging anything, and the area under the ROC curve.
import Foundation
func six(_ x: Double) -> String { String(format: "%.6f", x) }
// Read defective,score lines into tuples. split skips empty pieces, so a
// blank last line doesn't become a row. Double(_:) gives an optional, which
// ! unwraps.
let text = try String(contentsOfFile: CommandLine.arguments[1], encoding: .utf8)
var parts: [(defective: Bool, score: Double)] = []
for line in text.split(separator: "\n").dropFirst() { // dropFirst skips the header
let fields = line.split(separator: ",")
parts.append((defective: fields[0] == "1", score: Double(fields[1])!))
}
let n = Double(parts.count)
let defectiveCount = parts.filter { $0.defective }.count
// Sort every part into one of four counts at this threshold, then print the
// counts and the ratios built from them.
func report(_ suffix: String, _ threshold: Double) {
var tp = 0, fp = 0, fn = 0, tn = 0
for part in parts {
let flagged = part.score >= threshold // flagged when the score reaches it
switch (flagged, part.defective) {
case (true, true): tp += 1
case (true, false): fp += 1
case (false, true): fn += 1
case (false, false): tn += 1
}
}
// Swift never mixes Int and Double silently, so each count is converted.
let accuracy = Double(tp + tn) / n
let precision = Double(tp) / Double(tp + fp) // of the flagged, how many defective
let recall = Double(tp) / Double(tp + fn) // of the defective, how many flagged
let f1 = 2 * precision * recall / (precision + recall)
print("tp_at_\(suffix)", tp)
print("fp_at_\(suffix)", fp)
print("fn_at_\(suffix)", fn)
print("tn_at_\(suffix)", tn)
print("accuracy_at_\(suffix)", six(accuracy))
print("precision_at_\(suffix)", six(precision))
print("recall_at_\(suffix)", six(recall))
print("f1_at_\(suffix)", six(f1))
}
// The chance that a random defective part outscores a random good one.
// Compare every pair, scoring 2 for a win and 1 for a tie, so the count stays
// a whole number until the final division.
func auc() -> Double {
let defective = parts.filter { $0.defective }.map { $0.score }
let good = parts.filter { !$0.defective }.map { $0.score }
var points = 0
for d in defective {
for g in good {
if d > g {
points += 2
} else if d == g { // equal text in the file parses to equal doubles
points += 1
}
}
}
return Double(points) / Double(2 * defective.count * good.count)
}
print("parts", parts.count)
print("defective", defectiveCount)
// Never flagging anything is right about every good part.
print("baseline_accuracy", six(Double(parts.count - defectiveCount) / n))
report("50", 0.5)
report("30", 0.3)
print("auc", six(auc()))Each row becomes a named tuple of a Bool and a Double, so we skip
declaring a struct. report switches on the pair (flagged, defective),
and the compiler checks that the four cases cover every combination, one
per cell of the confusion matrix.
Swift won’t multiply or divide an Int by a Double: a * b with one of
each is a compile error, so every count goes through Double(_:) before a
ratio. For the AUC, filter and map pull out the defective and good
scores, and two nested loops compare every pair, 2 points for a win and 1
for a tie, as a whole-number Int until the last division.
# Evaluation metrics for a binary classifier, using only Tcl's built-in
# commands.
#
# Reads each part's true label and the model's score, then reports the
# confusion matrix, accuracy, precision, recall, and F1 at two thresholds,
# the accuracy of never flagging anything, and the area under the ROC curve.
# Returns a list of parts, each a two-element list {defective score}, where
# defective is 1 or 0.
proc read_parts {path} {
set f [open $path r]
set header [split [gets $f] ,]
set defective_col [lsearch -exact $header defective]
set score_col [lsearch -exact $header score]
if {$defective_col < 0 || $score_col < 0} { error "column not found" }
set parts {}
while {[gets $f line] >= 0} {
set fields [split $line ,]
lappend parts [list [lindex $fields $defective_col] [lindex $fields $score_col]]
}
close $f
return $parts
}
proc report {parts suffix threshold} {
# A part is flagged when its score reaches the threshold.
set tp 0; set fp 0; set fn 0; set tn 0
foreach part $parts {
lassign $part defective score
set flagged [expr {$score >= $threshold}]
if {$flagged && $defective} {
incr tp
} elseif {$flagged} {
incr fp
} elseif {$defective} {
incr fn
} else {
incr tn
}
}
# double() makes / divide as a decimal; between two integers, / rounds down.
set accuracy [expr {double($tp + $tn) / [llength $parts]}]
set precision [expr {double($tp) / ($tp + $fp)}] ;# of the flagged parts, how many are defective
set recall [expr {double($tp) / ($tp + $fn)}] ;# of the defective parts, how many are flagged
set f1 [expr {2 * $precision * $recall / ($precision + $recall)}]
puts "tp_at_$suffix $tp"
puts "fp_at_$suffix $fp"
puts "fn_at_$suffix $fn"
puts "tn_at_$suffix $tn"
puts "accuracy_at_$suffix [format %.6f $accuracy]"
puts "precision_at_$suffix [format %.6f $precision]"
puts "recall_at_$suffix [format %.6f $recall]"
puts "f1_at_$suffix [format %.6f $f1]"
}
# The chance that a random defective part outscores a random good one.
# Compare every pair, scoring 2 for a win and 1 for a tie, so the count
# stays a whole number until the final division.
proc auc {parts} {
set defective {}
set good {}
foreach part $parts {
lassign $part is_defective score
if {$is_defective} { lappend defective $score } else { lappend good $score }
}
set points 0
foreach d $defective {
foreach g $good {
if {$d > $g} {
incr points 2
} elseif {$d == $g} {
incr points
}
}
}
return [expr {double($points) / (2 * [llength $defective] * [llength $good])}]
}
set parts [read_parts [lindex $argv 0]]
set n [llength $parts]
set defective 0
foreach part $parts { incr defective [lindex $part 0] }
puts "parts $n"
puts "defective $defective"
# Never flagging anything is right about every good part.
puts "baseline_accuracy [format %.6f [expr {double($n - $defective) / $n}]]"
report $parts 50 0.5
report $parts 30 0.3
puts "auc [format %.6f [auc $parts]]"Each part is a two-element list of a label and a score, and lassign
unpacks it into two variables. One loop per threshold sorts every part
into one of four counts: true positives, false positives, false negatives,
and true negatives. Every metric is a ratio of those counts.
The scores stay strings, as everything in Tcl does, and expr compares
them as numbers when they look like numbers, so ties stay ties. Between two
integers, / rounds down, so the ratios use double() first. Tcllib
isn’t installed.