Visokio website     Downloads     Video tutorials     KnowledgeBase  
Idea: Scheduler - Running 2+ jobs at one time - Visokio Forums
Idea: Scheduler - Running 2+ jobs at one time
  • VoteVote Up9Vote Down     neilmerchant June 30, 2011 5:11AM
    Hi - I was wondering how you run 2 jobs at once in the scheduler? We have a job that take 4 hours and it blocks everything else from running during this time. The server is powerful enough to do this. I assumed it was using "fork execution" but that does not work. - Thanks Neil
  • 20 Comments
  •     steve June 30, 2011 8:58AM
    Neil, "fork" merely controls whether the scheduler runs tasks in the same process as the scheduling logic or spawns a separate process. It does not affect sequencing of tasks.

    Currently the scheduler only executes tasks in series. The watch folder option also executes tasks in series. However you can execute tasks in the scheduler and the watch folder concurrently (so two at once) but I'm presuming this is too cumbersome.

    It may be possible to have two schedulers running simultaneously with separate configuration files. This would help, I think. I'll post back here later today with more details.
  •     neilmerchant June 30, 2011 9:04AM
    Hi Steve - thanks for confirming. Before you spend to much time I under estimated the 2+, I am talking more 20 to 50+ We have some massive jobs to process!
  •     steve June 30, 2011 10:02AM
    OK, while it might be possible to configure multiple independent schedulers with different configuration files (and I won't spend any time investigating this - thanks), what you're after is clearly the ability to run an unlimited number of concurrent tasks within the same scheduler instance, either via the watch folder or via the internal scheduling system.

    One reason this hasn't been developed to date is for due to memory management. Omniscope doesn't have the sophistication to accurately model how much memory is needed for each task, so cannot safely work out how many tasks can be run in parallel. But it would be helpful to give you the ability to make that judgement yourself.

    I've reclassified this as an idea. In detail, the scheduler configuration would allow you to specify the concurrency level (as a maximum number of concurrent tasks). It would be up to the (power) user to manage the number of tasks and their memory requirements.
    In addition, perhaps you would be able to specify additional rules, such as "don't execute task X in parallel" or "don't start a new task until at least X memory is free".
  •     neilmerchant June 30, 2011 10:51AM
    Hi Steve
    This is exactly what I am after! Having a big job run by itself, then a load of smaller tasks starting once that main task has done. As you say I probably wouldn't want to run 20 jobs at once but maybe 5 and then start job six when any of 1-5 has finished.

    My next question is... when can this be done :)

    Regards
    Neil
  •     ygulla November 2, 2011 7:14AM
    The ability to execute tasks in parallel would be extremely useful for us. We have some reports that have an undefined time of running (due to the datasource limitations, nothing to do with Omniscope itself). They clog up the pipeline while not taking much processing power. So definitely a thumbs up from me!
  •     bfromson1 November 2, 2011 8:00AM
    And another thumb from me!
  •        benjamin November 2, 2011 8:12AM
    and me
  •     brian.lau November 2, 2011 9:57AM
    Definitely beneficial to have paraellel scheduling. This would be great help.
  • edtclarke     edtclarke November 7, 2011 11:13AM
    In the meantime it might be a workaround if you can run the tasks via the command line. We have been testing running multiple tasks at once and it seems to work well
    Atheon Analytics Ltd
    w: www.atheonanalytics.com
    e: ed.crawford@atheon.co.uk
    t: +44 8444 145501
    m: +44 7789 393 087
    s: edtclarke
  •     steve November 7, 2011 12:29PM
  •     bgrovens January 31, 2012 10:41AM
    Hi All,

    I have added a vote for this functionality as well.

    Regards,
    Brian
  •     rajeshbalu29 February 1, 2012 7:23PM
    Hi All,

    I do have added a vote for this functionality as well...

    Regards,
    Rajesh
  • Alexander     Alexander April 12, 2012 3:21AM
    Hi Steve,

    What is the status of this topic?

    Thanks!
  • Alexander     Alexander April 12, 2012 5:43AM
    ok. Thanks. Can you help me setting up a second scheduler?
    I need one task to run at least every 30 mins. Tasks that take longer then 30 mins prevent this.

    Is skype the fastest solution?
  •     steve April 12, 2012 7:10AM
    We'll post instructions shortly, and take it from there.
  • Alexander     Alexander April 12, 2012 7:14AM
    thanks!
  •     michael April 13, 2012 11:15AM
    Hi,

    Here is the description how you can configure "Running multiple Scheduler instances concurrently"
    http://forums.visokio.com/discussion/1469/running-multiple-scheduler-instances-concurrently
  •     steve March 24, 2015 11:02AM
    Alpha partners, concurrent execution has now been integrated into "2.9 Plus"; see here:
    http://forums.visokio.com/discussion/2867/scheduler-concurrent-task-execution

Welcome!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership