Scheduler: task execution speed?

mburgess · September 6, 2012 10:27AM

We've been using the Scheduler a while now and have noticed some of our files are performing much slower than when manually updated. The iok's in question have two tasks performed on them

1. Refresh from Source
2. Publish

...then save (which does not take long)

My question on Scheduler is do these two tasks repeat the whole DataManager process - i.e refresh from source loads the data, runs the DataManager process, then loads data set into Omniscope (like refresh icon in top right of Omniscope?

Whereas publish in Scheduler will load the data, run the DataManager process and then publish to csv?

So, for a file where we only need to publish do we need to 'Refresh from source' in the Scheduler as well?

Thanks -Matt

steve · September 6, 2012 11:38AM

I suspect the problem is that Omniscope on the desktop isn't clearing the disk cache, whereas the Scheduler always does, every task, to guarantee latest data in a publishing situation.

In Settings > Advanced, enable "Show diagnosis tools".

Now edit your scheduler configuration. You should now see "Clear disk cache" options in your existing action configurations which you should untick. I recommend adding a single manual "Clear Disk Cache" action before the entire workflow.

You can also tick "Log timings" in the File action task to measure performance accurately.

mburgess · September 6, 2012 11:49AM

Steve - could you just answer:

For a file where we only need to 'publish' do we need to 'refresh from source' in the Scheduler as well? (we only want the published files not what is loaded into Omniscope)

thanks

steve · September 7, 2012 4:37AM

If your DataManager workflow involves a series of sources, operations and ultimately one or more publishers, and *doesn't* involve the "Omniscope" block, then you definitely do *not* need to refresh from source. In this case you are using DataManager as an ETL tool completely independently from the Omniscope views and tabs.

"Refresh from source" means "load the latest data into the Omniscope report".

Welcome!

Categories

Ideas Parade

Tagged