Extract line from Sf bench command

[ChessDrone]

Cosmetic changes

Code: Select all

#!/bin/bash

# http://www.open-chess.org/viewtopic.php?f=5&t=2557#p20321 modified by wmahan

for engine in stockfish*; do
    div=50
    sum=0
    average=0
    for (( i = 1; i <= $div; i++ ))
    do
        out=`cat <<END | $engine 2>&1 | tail
setoption name Idle Threads Sleep value false
isready
bench
END`
        nps=`echo "$out" | grep "Nodes\/second"|sed "s/.*: \([0-9]*\)/\1/"`
        nodes=`echo "$out" | grep "Nodes searched"| sed "s/.*: \([0-9]*\)/\1/"`
        let sum=$sum+$nps
    done
    let average=$sum/$div ; echo -e "$engine\t$nodes\t$average\r">>results.csv
done

[quantum]

I did not look into this forum for a longer time and just came across your changes. Based on this the script now checks if the nodes searched are consistent. Automatic calculation of the standard deviation is also included. For the moment it works, but I think it should be cleaned up at some point.

Code: Select all

#!/bin/bash

# http://www.open-chess.org/viewtopic.php?f=5&t=2557#p20321 
# From quantum, lucasart and wmahan.

for engine in ./stockfish*; do
    div=2 #must be >1 otherwise we divide by zero later
    sum=0
    average=0
    stdev=0
    summe=0
    var=0
    changes=0
    oldnodes=0
    

    declare -a number

    for (( i = 1; i <= $div; i++ ))
    do
        out=`cat <<END | $engine 2>&1 | tail
setoption name Idle Threads Sleep value false
isready
bench 32 1 13 default depth
END`
        nps=`echo "$out" | grep "Nodes\/second"|sed "s/.*: \([0-9]*\)/\1/"`
        nodes=`echo "$out" | grep "Nodes searched"| sed "s/.*: \([0-9]*\)/\1/"`

#Check if nodes searched are the same.
	if [ "$oldnodes" != 0 ]&&[ "$oldnodes"  != "$nodes" ]; then
 consistency="Nodes searched are not consistent!"
fi
	oldnodes=$nodes

#Calculate average nps
        let sum=$sum+$nps
	number=("${number[@]}" "$nps")
    done
    let average=$sum/$div

#Calculate standard deviation
    for (( i = 1; i <= $div; i++ ))
    do  
        summe=0
	summe=$(echo "(${number[n]}-$average)^2" | bc -l)
        let var=$var+$summe
    done
	stdev=$(echo "scale=0;sqrt($var/($div-1))" | bc -l)

#Output
 echo -e "$engine\t$nodes\t$average\t$stdev\t$consistency\r">>results.csv
done

[quantum]

Sorry for second post in a row. Did not find the edit button.

I just found out that the calculation of the standard deviation does not work correctly. Probably an issue with reading the values from the array.

[quantum]

Third post in a row as I am not supposed to edit my previous post. This version should calculate the standard deviation correctly.

I just benchmarked Sf 20 times and received following nps:

935006 923019 947309 941117 953582 941117 911335 935006 959940 935006 947309 928974 905603 899943 911335 911335 894354 959940 953582 935006

I wondered why several of these entries are the same???

Code: Select all

#!/bin/bash

# http://www.open-chess.org/viewtopic.php?f=5&t=2557#p20321 
# From quantum, lucasart and wmahan.

for engine in ./stockfish*; do
    div=20 #must be >1 otherwise we divide by zero later
    sum=0
    average=0
    stdev=0
    summe=0
    var=0
    changes=0
    oldnodes=0
    

    declare -a number

    for (( i = 1; i <= $div; i++ ))
    do
        out=`cat <<END | $engine 2>&1 | tail
setoption name Idle Threads Sleep value false
isready
bench 32 1 6 default depth
END`
        nps=`echo "$out" | grep "Nodes\/second"|sed "s/.*: \([0-9]*\)/\1/"`
        nodes=`echo "$out" | grep "Nodes searched"| sed "s/.*: \([0-9]*\)/\1/"`

#Check if nodes searched are the same.
	if [ "$oldnodes" != 0 ]&&[ "$oldnodes"  != "$nodes" ]; then
 consistency="Nodes searched are not consistent!"
fi
	oldnodes=$nodes

#Calculate average nps
        let sum=$sum+$nps
	number=("${number[@]}" "$nps")
    done
    let average=$sum/$div

#Calculate standard deviation
    for i in "${number[@]}"
    do  
        summe=0
	summe=$(echo "(${i}-$average)^2" | bc -l)
        let var=$var+$summe
    done
	stdev=$(echo "scale=0;sqrt($var/($div-1))" | bc -l)

#Output
 echo -e "$engine\t$nodes\t$average\t$stdev\t$consistency\r">>results.csv
done

[ChessDrone]

Thanks a lot, currently I try it.
For Windows, this will be needed: http://gnuwin32.sourceforge.net/packages/bc.htm
then paste it in the folder "bin" of the 'Bash' shell
Otherwise you'll get this error:
#./Bench.sh: line 45: bc: command not found
#./Bench.sh: line 46: let: var=0+: syntax error: operand expected (error token is "+")

I also bring an additional comment for the command Bench.
# bench 32 1 6 default depth
# bench (hash: n in mb) (threads: n) (limit: n (cf. limit type)) (FEN: default) (limit type: n of time (in ms?), n of nodes, n? mate or n of depth)

Edit: I only get 4 columns instead of 5, which is missing?

Code: Select all

profile-build/stockfish_profile-build_x32-473-posix-sjlj_x86-32.exe	144276	1031044	24041	
profile-build/stockfish_profile-build_x32-473-win32-sjlj_x86-32.exe	144276	1025696	104502	
profile-build/stockfish_profile-build_x32-481-win32-dwarf_x86-32.exe	144276	1016974	110381	
profile-build/stockfish_profile-build_x32-481-win32-sjlj_x86-32.exe	144276	1006293	118165	
profile-build/stockfish_profile-build_x64-473-posix-sjlj_x86-64-modern.exe	144276	1422040	833405	
profile-build/stockfish_profile-build_x64-473-win32-sjlj_x86-64-modern.exe	144276	1491265	979989	
profile-build/stockfish_profile-build_x64-481-posix-seh_x86-64-modern.exe	144276	1452021	901084	
profile-build/stockfish_profile-build_x64-481-posix-sjlj_x86-64-modern.exe	144276	1376998	765594	
profile-build/stockfish_profile-build_x64-481-win32-seh_x86-64-modern.exe	144276	1459870	925403	
profile-build/stockfish_profile-build_x64-481-win32-sjlj_x86-64-modern.exe	144276	1404150	820381	

[ChessDrone]

Ah ok, it is consistency.
Can you enlighten me on the Standard deviation? If the number is short, it is better?

[quantum]

If you run Sf bench command in multi core mode the nodes searched are not consistent. In the past it happened that a patch was introduced by mistake where the nodes searched were not consistent even in single core mode from time to time. Therefore I fought this future might be nice to have.

English is not my mother tongue nor am I statistician although I had some lessons in it...

To my understanding the standard deviation sigma gives you a hint how reliable your measurment was.

With a probability of 68% the true value is found to be within the range of your average value plusminus 1 sigma.
With a probability of 95 % the true value is found to be within the range of your average value plusminus 2 sigma.

For most purposes it is sufficient to use the 2sigma intervall as error bars.

I did run bench with different compilers.

./stockfish-clang 8347121 1088932 6168
./stockfish-clang-pgo 8347121 1091656 6941
./stockfish-gcc 8347121 1067495 37877
./stockfish-gcc-pgo 8347121 1087086 31128
./stockfish-icc 8347121 1048282 79075
./stockfish-icc-pgo 8347121 1063484 54086

Although it seems that the stockfish-icc compile is the slowest it could still be the fastest because the error bars are high. More testing is needed to reduce the error bars and to decide which compile is fastest.

All these error bars can trick you if you have a systematic error in your testing. For example you want to know the size of your room properly and do a sample of 20 measurments. Unfortunately the scale of your ruler is wrong. In this case you would get little error bars but your result is terribly wrong.

I hope that my expanation on this is not totally off...

[ChessDrone]

Thank you for the explanation. I understand a little better.

I ask if it is possible to make the "Bench" equivalent in Shell for engines such as Senpai who do not have this command. I would like to compare different compiler with Senpai.

[ChessDrone]

The command sequence equivalent to "bench 32 1 6 default depth" should be this:

Code: Select all

setoption name hash value $1
setoption name threads value $2
isready
position fen $4
go $5 $3

positions used by stockfish

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 10
8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 11
4rrk1/pp1n3p/3q2pQ/2p1pb2/2PP4/2P3N1/P2B2PP/4RRK1 b - - 7 19
rq3rk1/ppp2ppp/1bnpb3/3N2B1/3NP3/7P/PPPQ1PP1/2KR3R w - - 7 14
r1bq1r1k/1pp1n1pp/1p1p4/4p2Q/4Pp2/1BNP4/PPP2PPP/3R1RK1 w - - 2 14
r3r1k1/2p2ppp/p1p1bn2/8/1q2P3/2NPQN2/PPP3PP/R4RK1 b - - 2 15
r1bbk1nr/pp3p1p/2n5/1N4p1/2Np1B2/8/PPP2PPP/2KR1B1R w kq - 0 13
r1bq1rk1/ppp1nppp/4n3/3p3Q/3P4/1BP1B3/PP1N2PP/R4RK1 w - - 1 16
4r1k1/r1q2ppp/ppp2n2/4P3/5Rb1/1N1BQ3/PPP3PP/R5K1 w - - 1 17
2rqkb1r/ppp2p2/2npb1p1/1N1Nn2p/2P1PP2/8/PP2B1PP/R1BQK2R b KQ - 0 11
r1bq1r1k/b1p1npp1/p2p3p/1p6/3PP3/1B2NN2/PP3PPP/R2Q1RK1 w - - 1 16
3r1rk1/p5pp/bpp1pp2/8/q1PP1P2/b3P3/P2NQRPP/1R2B1K1 b - - 6 22
r1q2rk1/2p1bppp/2Pp4/p6b/Q1PNp3/4B3/PP1R1PPP/2K4R w - - 2 18
4k2r/1pb2ppp/1p2p3/1R1p4/3P4/2r1PN2/P4PPP/1R4K1 b - - 3 22
3q2k1/pb3p1p/4pbp1/2r5/PpN2N2/1P2P2P/5PP1/Q2R2K1 b - - 4 26
6k1/6p1/6Pp/ppp5/3pn2P/1P3K2/1PP2P2/3N4 b - - 0 1
3b4/5kp1/1p1p1p1p/pP1PpP1P/P1P1P3/3KN3/8/8 w - - 0 1
2K5/p7/7P/5pR1/8/5k2/r7/8 w - - 0 1
8/6pk/1p6/8/PP3p1p/5P2/4KP1q/3Q4 w - - 0 1
7k/3p2pp/4q3/8/4Q3/5Kp1/P6b/8 w - - 0 1
8/2p5/8/2kPKp1p/2p4P/2P5/3P4/8 w - - 0 1
8/1p3pp1/7p/5P1P/2k3P1/8/2K2P2/8 w - - 0 1
8/pp2r1k1/2p1p3/3pP2p/1P1P1P1P/P5KR/8/8 w - - 0 1
8/3p4/p1bk3p/Pp6/1Kp1PpPp/2P2P1P/2P5/5B2 b - - 0 1
5k2/7R/4P2p/5K2/p1r2P1p/8/8/8 b - - 0 1
6k1/6p1/P6p/r1N5/5p2/7P/1b3PP1/4R1K1 w - - 0 1
1r3k2/4q3/2Pp3b/3Bp3/2Q2p2/1p1P2P1/1P2KP2/3N4 w - - 0 1
6k1/4pp1p/3p2p1/P1pPb3/R7/1r2P1PP/3B1P2/6K1 w - - 0 1
8/3p3B/5p2/5P2/p7/PP5b/k7/6K1 w - - 0 1