BME680 API calibration calculation inconsistency

I found inconsistency between BME680 API and Datasheet for the calculation of gas sensor heater resistance.

Page 21 of Datasheet explains floating point and integer calculations but they are not equivalent. var1 in integer calculation is (amb_temp*par_g3/10)<<8, but the floating point method calculates this part as (amb_temp*par_g3*10)<<8 (var1 in the floating point method is different from that in the integer method, so I deduced this result by rearranging the whole floating point formulae).

Moreover, the integer calculation in API is var1=(amb_temp*par_g3)/1000)*256 (small modifications applied for clarity) in line 1006 of bme680.c which is again different from the above two. The floating point calculation in API looks the same as that in Datasheet.

So, there are three variations in calculating var1 or its equivalent, each 100 times different from another. Could you tell me which is correct, and hopefully fix the incorrect formulae? Thank you.

Best reply by Vincent

Thank you.  i will double check with the API team and also the datasheet then come back to you

View original
3 replies
Resolved