After a think,

I guess the answer is that the error bars refers to the test suite used. I once created a 4-ply 50 positions one (i.e 1. e4 e5 2. f4 ef4, 1.e4 e5 f4 Qh4+, 1.d4 f5 2. c4 g6 etc...) and I noticed important variation in the results (unfortunately I don't have these any more since OS ...

Thank you for your quick answer BB+,

I'm not a programmer, but I'm my "testing" (for fun of course), I used to use 50 positions test suites. I noticed two things:

1- The multinomial thing, you just mentioned you don't think is much important (thank you for your quick answer).

2- The test suites ...

I'm not strong enough in statistics to answer my question but there it is: it appears to me that when we are using "test suite" (the engines play the same position twice, with side reversed), we should used a multinomial distribution (i.e. scores will be either 0/2, 0,5/2, 1/2, 1,5/2 or 2/2 per ...