Force refresh of IOK file within DM

May 20, 2010 5:10PM

In order to tackle complex data build processes it may be advantageous to divide a task into a number of sub-tasks - for example, in building up a number of data sets from a range of spreadsheets in different directories, it may be useful to treat each directory independently (especially if each handles different types/formats of data). It seems sensible to build a DM IOK file to handle each sub-task, and then build a final DM IOK which uses each sub-task IOK as a data source. This is easy to do, but opening and publishing the 'master' DM IOK doesn't force each of the sub-task IOK files to rebuild itself...

... it would be good if, when handling any IOK file as a data source, there was an option to force the IOK data source to rebuild itself.

steve · May 20, 2010 6:44PM

I presume you're using the latest 2.6. If you save the sub-task IOK files with "refresh on open = always" configured, does this solve the problem?

May 20, 2010 8:45PM

Steve

Apologies - yes, it does appear to... I'm just letting it run over a directory of 15 9-worksheet XLS files, lots of flashing of the mouse pointer so I think it works fine :)

Guy

steve · May 21, 2010 6:25AM

By the way, you should experiment with the option *not* to automate Excel when importing XLS files.

2.6 has two Excel file reader libraries, one of which (from 2.5 and earlier) starts Excel in the background and "reads" cells from it; the other is new in 2.6 and reads XLS / XLSX files directly in Java. This is much faster (and also completely cross-platform, i.e. available on Mac), but causes a significant memory spike (released after opening the file).

We've kept the former as the default for compatibility reasons, since the newer one may have differences of behaviour and won't support 3rd party formula functions; there may be other limitations we haven't yet discovered. But if the spreadsheets you're reading don't use installed additional formula functions (for example) and you have 500M+ spare in the JVM (this is an estimate for large spreadsheets), try unticking "Excel" in the data source block or the "customise data import" dialog.

May 21, 2010 8:10AM

OK - will try this today (lots more Excel processing to do) and will supply some feedback...

Welcome!

Categories

Ideas Parade