I have data sets of identical data for multiple periods (e.g., years) with about 150 columns and 50,000 rows each. Each time data for a new period is available, my plan was to copy the old period file and then import the new data into the "current" period file, because it has my views set up in tabs and because I only need to display details for the current period. I have another database table where, for a few variables, I track trending information (the value of a variable) for each period. I'd like to display that trending information along with the data for the current period in a few select tabs. I'm guessing that I should use the approach described in the thread on here about using multiple data sets in the same IOK file, is that right? I'm also guessing that performance would be impacted quite a bit if I appended the data for each new period, am I right? If you have any other creative solutions I'm all ears. If you want to see a picture of what I'm talking about, I did a cut and paste that you can view at http://sample.swmcdonnell.com. Thanks in advance for any suggestions or advice.
To be clear:
- the 150x50,000 granular period data is, let's say, for 1 year.
- you have multiple years, but only show this granular period data for a single period (year)
- you also have a separate table of trending data (e.g. an aggregation of multiple years).
Is this right?
I'm glad you've discovered you can re-import data from different periods into a "template" file. If you try and combine multiple periods of granular data together, this should work, but if you have many periods, it will begin to require a beefier computer (64-bit with 4 or more GB memory).
I would have thought, however, that combining the trending data with a single period should have no real impact on performance. You would do this in the way suggested - concatenate the two files with different data structures, making sure the fields have new names; then have an additional field which is used to identify the table of data. Providing the trending data is much smaller, it should have no effect on performance.