Visokio website     Downloads     Video tutorials     KnowledgeBase  
Decimal place question - Visokio Forums
Decimal place question
  •     alec_webliquid June 11, 2012 9:31AM
    Hi,

    Question for the Omniscope team: when setting the number of decimal places for a field, is the field value stored with that number? Or is it stored with the full number of decimals (however many that may be), but displayed with the amount set (similar to Excel)? I ask because I have noticed some small discrepancies between values calculated live in a table view (formula result on aggregated data) and values I have calculated elsewhere from the raw data, in a single function call.

    Many thanks!

    EDIT: it is of course entirely possible my formulas are not quite correct, but the question still stands
  • 1 Comment
  •     steve June 11, 2012 10:25AM
    Omniscope stores numbers with 64-bit precision (around 17 significant figures), regardless of the number of decimal places configured in the field.

Welcome!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership

Tagged