i just had a question before spending time building a report
we have a data source we are getting daily in xls format - approx 7,000 rows x 5 fields
we are looking to use batch processing to create on table for a years (i.e. 365 files) data initially (and then continue to update going forward with future files)
would there be any issues using batch processing with this large amount of source files? is there a better way other than batch processing for Visokio to cope with this many data source files
There may be an issue with the total volume of data (~2 million records per year), depending on your PC configuration. You may need 8gb of memory or so.
You should use DataManager to batch append the files. This should take no time to set up and experiment with. If you find there are performance issues with this, let us know; you may be able to work around them by, for example, batch appending a month at a time, and saving the results as an IOK file, then batch appending the IOK files.