http://forums.visokio.com/discussions/tagged/accuracy/feed.rss Mon, 30 Oct 17 18:31:28 -0400 Tagged with accuracy - Visokio Forums en-CA http://forums.visokio.com/discussion/2601/display-displaying-full-15-decimal-place-accuracys Tue, 17 Jun 2014 11:05:45 -0400 acdewinter 2601@/discussions
I've run into a serious problem when working with numbers/decimals in Omniscope when pulling the data through SQL Server.

I was a little perplexed when I was dividing a large number by 100 and it kept rounding the output to 1 decimal place instead of two. So I went about making sure I use Convert(decimal) as well as Round to ensure I get both decimals. However it kept only returning the single decimal in a rounded format. I then tried several different data types and still sat with the same problem.

So I decided to run some tests by just doing my own calculations instead of pulling out specific fields (in case this was a DB problem)

A simple calculation of 100/3 which should return 33.3333333 returns only 33. So I converted both numbers to decimal and it returned: 33.33333
Success!

Then why did I still have a problem?

So then I tested 1000000/3 and low and behold it returns 333333.3 (only one decimal) and no matter what you do to it, it will not return a second decimal or more.

So then I tested 10000000/3 and it returned absolutely no decimals, which then lead me to the conclusion that you are limited to 7 characters when working with decimals in omniscope.

The size of the integer can keep increasing but you can't have decimals when it exceeds 7 character length.

I have attached a screenshot for your viewing to illustrate what I mean.

Please could someone help me with this, as the data I'm working with needs to be accurate to the decimals I specify.

Just incase someone suggests this: using Cast instead of Convert makes no difference, and infact works even less.

Edit:

I see another colleague has brought this up previously around aggregation blocks: http://forums.visokio.com/discussion/2437

It seems even with SQL it does retain the precision and you don't require to run conversions on all the individual parts. I just popped a field organiser on and it seems to work.

Maybe a suggestion going forward would be for you guys to add in a option to use/not use this function on a datasource and an option for the user?]]>