Scheduler automated data refresh: clearing the disk cache

Mees · August 9, 2011 1:45PM

Hi there,

Sometimes we notice that the final DM block indicates the full DM process is ready (by showing the Load button), while somewhere upstream some blocks are still running. Are you aware of this behaviour? And how can we prevent it?

Now we are using the server edition and we run some DM files in sequence in the scheduler. We have the feeling that some blocks haven't been completed because we miss some data in our final Omniscope files. A reason could be what I have described above: the final block was loaded in Omniscope too early. We can't check this however, as the scheduler runs these files in the background.

Can you please help as this is important for us. We must be able to trust the DM processes completely otherwise we are publishing incomplete information.

Many thanks

steve · August 9, 2011 2:27PM

The DataManager you see in Omniscope as a view uses asynchronous loading of data. So, if you have an append with 2 inputs, each input can refresh independently, and downstream blocks may prompt Load when the first input has finished, before waiting for the 2nd input to finish. Note also that, when an upstream block refreshes, it doesn't mean you will see new data; it just means that Omniscope is *looking* for changes.

The DataManager process in the scheduler does *not* use asynchronous loading. And unless you use some advanced diagnostic features, which you will know about if you are using, every DataManager action in the scheduler (such as "Refresh from source" or "Publish all output blocks") starts from scratch without any cached data. This is deliberately to guarantee the latest data, at the expense of longer workflow for repeated similar tasks.

I recently gave you details of those advanced features to speed up processing of a series of parameterised client files by reusing cached data across a sequence of operations. When you do this, data is only guaranteed to be as recent as the last time you executed a "clear disk cache" task. Make sure you have a "clear disk cache" task scheduled as often as necessary to ensure your data is up to date.

Please post back if you need more help with this.

Mees · August 11, 2011 4:50AM

Thanks Steve. Good to know about the asynchronous loading in DM vs Scheduler.
Process runs smooth at the moment and are playing with the "clear disk cache" tasks.
Thanks so far

Welcome!

Categories

Ideas Parade

Tagged