Visokio website     Downloads     Video tutorials     KnowledgeBase  
Refresh: Controlling 'upstream' refreshes when joining many files? - Visokio Forums
Refresh: Controlling 'upstream' refreshes when joining many files?
  •        benjamin February 18, 2015 11:31AM
    Hi

    I have a series of DataManager blocks and at near the end of the series i am joining various streams of blocks together by a merge

    Whenever i update the merge (either by changing the join criteria, adding fields etc) the whole upstream data gets refreshed in all the other blocks.

    Is there any way to stop this, i do not want it to refresh data when nothing has changed, I just want to update with my new fields in the join criteria

    Thanks
  • 8 Comments
  •     paola February 18, 2015 11:44AM
    Ben - Yes, if you are working on configuring a file for the first time, and the source data files are not changing, then you should turn off refresh of the Source blocks.

    You can set different refresh behaviour for each of the Source blocks using the blue spanner/wrench block options drop-down on the right side of the Source blocks. From the drop-down, choose Update options) and set it to "Never", while you're working on the workflow (other alternatives are Auto-detect and Scheduled periods). After completion/testing switch the Update options settings to something appropriate.
  •        benjamin February 18, 2015 12:14PM
    Hi Paola

    I have tried
    A.Unselecting background refresh in the file menue
    B.Selecting refresh to "Never" on the SQL block itself

    However in both cases when i open the file and add in another field in the right datamanager stream, when this stream then joins onto the left datamanager stream it updates the whole left hand side stream

  •     paola February 18, 2015 12:16PM
    Would you be able to post or email us directly a demo file?
    thanks
  •        benjamin February 18, 2015 12:29PM
    If you open the file and then click "Open with stale data"
    Now in Field organiser 4 if you add or delete a column
    You will then notice that the whole upstream block on the LHS updates

    It would be better example if you connected it to a database on the LHS that takes a long/slow time to query

  •     enrico68 June 25, 2015 9:37AM
    I have one question related to this subject.

    I have a source1.iok file that is the only input of a second source2.iok file.

    I save the source2.iok file with the above indicated 'never' option activated on the source1.iok input block and I prepare a scheduled process in which I refresh and save the source1.iok file and, later, I refresh and save the source2.iok file.

    Which is the behaviour of the scheduled process? Will the source2.iok file be correctly updated at the end of the scheduled process also with the 'never' option activated?

    Thanks
  •     tjbate June 26, 2015 7:40AM
    Enrico - Good question...

    When you have an IOK file (1) as the source of another IOK file (2), refreshing file (2) will always retrieve data from what is contained within file (1). The refresh option 'never' applies to whether you at the same time refresh data in (1) from its own source, which might typically be a CSV or Excel file.

    So typically you might have a data flow chain:
    CSV --> IOK1 --> IOK2

    If as stated, IOK2 has refresh option "Never" configured in its DataManager source block referencing the 'upstream' IOK1, and you schedule a "file action" upon IOK2 that refreshes from source and saves, it will read latest data from IOK1 and flow it through the IOK2 DM workflow. In this case, even if the original source CSV file has newer data, these changes will be ignored; the data set 'cached' in IOK1 will be used instead.

    If instead IOK2 has refresh option set to "Always", the same scheduled "file action" task will go all the way upstream and first read latest data from CSV file, then flow it through any DM sequence modelled in IOK1, before using the fully updated data in the IOK2 workflow. Hence the CSV latest data will be reflected correctly, and any stale 'cached' data in IOK1 will not be used.
  •     acohen September 11, 2015 12:51PM
    Hi, I'm having the same problem when trying to work on an Omniscope file with Upstream operations. This file has queries that take a very long time to run and I am trying to debug something downstream however whenever I try to look at what is happening downstream it triggers a full upstream refresh, paralyzing the work I am doing.

    In previous versions of Omniscope I would take a snapshot of a block and then feed the downstream area I am working on off of the snapshot. However when running 2.9 plus b1765, even snapshotting causes a full upstream refresh.
  •     paola September 14, 2015 12:56PM
    Hi, we could not reproduce this behaviour, but would suggest that you take a snapshot of the first block below (Append or Merge ) and then set its' refresh behaviour to "Never". This should definitely cut off the connection with the database, while preserving the data.

Welcome!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership