Hi - I was wondering how you run 2 jobs at once in the scheduler? We have a job that take 4 hours and it blocks everything else from running during this time. The server is powerful enough to do this. I assumed it was using "fork execution" but that does not work. - Thanks Neil
Neil, "fork" merely controls whether the scheduler runs tasks in the same process as the scheduling logic or spawns a separate process. It does not affect sequencing of tasks.
Currently the scheduler only executes tasks in series. The watch folder option also executes tasks in series. However you can execute tasks in the scheduler and the watch folder concurrently (so two at once) but I'm presuming this is too cumbersome.
It may be possible to have two schedulers running simultaneously with separate configuration files. This would help, I think. I'll post back here later today with more details.
Hi Steve - thanks for confirming. Before you spend to much time I under estimated the 2+, I am talking more 20 to 50+ We have some massive jobs to process!
OK, while it might be possible to configure multiple independent schedulers with different configuration files (and I won't spend any time investigating this - thanks), what you're after is clearly the ability to run an unlimited number of concurrent tasks within the same scheduler instance, either via the watch folder or via the internal scheduling system.
One reason this hasn't been developed to date is for due to memory management. Omniscope doesn't have the sophistication to accurately model how much memory is needed for each task, so cannot safely work out how many tasks can be run in parallel. But it would be helpful to give you the ability to make that judgement yourself.
I've reclassified this as an idea. In detail, the scheduler configuration would allow you to specify the concurrency level (as a maximum number of concurrent tasks). It would be up to the (power) user to manage the number of tasks and their memory requirements. In addition, perhaps you would be able to specify additional rules, such as "don't execute task X in parallel" or "don't start a new task until at least X memory is free".
Hi Steve This is exactly what I am after! Having a big job run by itself, then a load of smaller tasks starting once that main task has done. As you say I probably wouldn't want to run 20 jobs at once but maybe 5 and then start job six when any of 1-5 has finished.
The ability to execute tasks in parallel would be extremely useful for us. We have some reports that have an undefined time of running (due to the datasource limitations, nothing to do with Omniscope itself). They clog up the pipeline while not taking much processing power. So definitely a thumbs up from me!
In the meantime it might be a workaround if you can run the tasks via the command line. We have been testing running multiple tasks at once and it seems to work well
ok. Thanks. Can you help me setting up a second scheduler? I need one task to run at least every 30 mins. Tasks that take longer then 30 mins prevent this.