Aggregation: decimal number sum precision
-
I just found some strange decimal number behaviour in Omniscope 2.8 b1190
It looks like when aggregating 2 decimal numbers with more than 7 digits the sum loses some precision in an unexpected manner:
e.g. in the attached example 100,000.51 + 0.11 = 100,000.6 (see “loss of precision” block)
The rest of the blocks just show some examples that work.
Attachments
Aggregation Precision.iok 10K -
1 Comment
-
Thanks for sending this. Omniscope will by default round the result to 7 significant figures, in effort to avoid too many decimals in some common cases.
The value is preserved, and you can add the FieldOrganiser either before or after the Aggregation block to regulate the outcome.
Welcome!
It looks like you're new here. If you want to get involved, click one of these buttons!
Categories
- All Discussions2,595
- General680
- Blog126
- Support1,177
- Ideas527
- Demos11
- Power tips72
- 3.0 preview2
To send files privately to Visokio email support@visokio.com the files together with a brief description of the problem.