Blog

Menus: Simpler "Advanced settings" (2.8+)

steve — Wed, 15 Aug 2012 12:58:47 -0400

From tonight, the "Settings > Advanced" menu has been restructured in 2.8. Power users who access settings in these menus should find them much easier to navigate.

Note that the "Performance" section (containing very rarely needed advanced settings) is only shown if "Settings > Advanced > Feature availability > Debugging, diagnostics and performance" is enabled.

Scheduler: Import/export/copy/paste actions, tasks & recurrences (2.8+)

steve — Tue, 31 Jul 2012 10:45:19 -0400

Tonight's build of 2.8 brings some simple time-saving improvements to the Scheduler UI.

You can now import/export/copy/paste any actions, tasks and recurrences.

From 2.8 b226 onwards.

Formulae: new TRENDVALUE function (2.8+)

steve — Thu, 02 Aug 2012 09:57:46 -0400

In tonight's build of 2.8, the new TRENDVALUE function has been added.

This allows you to work out the Y value for a line of best fit (linear trend) given any two numeric/date fields. This is equivalent to what you see in the Graph view when you show a line of best fit.

For an example, see here:
http://forums.visokio.com/discussion/comment/6116/#Comment_6116

Selection: Filters brushing (2.8+)

steve — Wed, 01 Aug 2012 13:29:08 -0400

For a long time now Omniscope has supported "Selection brushing" (previously called just "Brushing").

With selection brushing, when you select data in one view, other views update to show the same data, with a shadow of the original data shown behind.

Omniscope 2.8 now supports "Filter brushing", available via opt-in to any views configured to show "All data". With this, as you filter, the view updates to show the filtered data, with a shadow of the original data. See example below.

Selection brushing is enabled by default. Filter brushing is not.

To explore filter brushing in a given view:

Choose View toolbar > Subset > All data
Choose View toolbar > Tools > Brushing > Filters brushing

Note that selection and filters brushing are now configurable at a per-view level (as above) as well as a global level (Main toolbar > Settings > Brushing)

Please note that this feature is still in development and may evolve further in the coming days.

Example:

Scripting: Using arrays of data (2.8+)

steve — Mon, 16 Jul 2012 11:23:32 -0400

Version 2.8, available shortly to alpha partners, introduces a new function "dataArray", for use in Javascript. It retrieves a two-dimensional array of cell data for bulk processing and transformation.

You can use Javascript in Omniscope in the SCRIPT function in formulae, and to do HTML pre-processing in the Content view.

Syntax:

dataArray(fields, rowSubset, includeFieldNames)

Argument "fields": An optional array of field names. If omitted, all fields are retrieved.
Argument "rowSubset": An optional data subset, as returned by the subset function.
Argument "includeFieldNames": if true, the first 'row' in the returned 2d array 'table' will be the field names.

Returns: A two-dimensional array of cell data, indexed as [column][row], with indexes numbered from 0 to n-1.

See attached example, using dataArray inside a Content View.

Content View: Custom scripting (2.7+)

steve — Fri, 28 Oct 2011 12:43:12 -0400

This new feature allows scripting-savvy publishers to write JavaScript inside their Content View in "view source" mode to create dynamic, adaptive text. This is very similar to Java Server Pages (JSP).

For example, this allows you to repeat a section of your document for different subsets of the data.

Scroll down for an example IOK file.

Requires 2.7 b96 or later. You must use "View Source" mode (the "<>" button in the Content View toolbar).

Existing syntax for dynamic content - {{ }}

The existing dynamic content syntax is as follows. You might not use this, but if you have inserted dynamic content using the "f" button in the Content View, this is what happens in "view source" mode:

Total: {{ SUBSET_SUM("Coupon") }}

This example inserts the dynamic result of the sum of the "Coupon" field according to whatever data results after applying your filters.

New syntax for scripting - <# #> and <#= #>

<#
  if (x==5) {
    y = 3;
  } else {
    y = 2;
  }
  out.println("Write some text into the document, programmatically");
  var array = subset_uniquesList("Category");
  for (var i in array) {
    #>
    Some text and HTML with <#=array[i]#> (inserted values)
    <#
  }
#>

This example demonstrates some JavaScript then uses a loop over unique values (after applying filters) in a field to repeat a section containing both static content and inserted script expressions.

See the attached file for a larger example including comments.

Functions available in script blocks

formula("IF(RAND<0.5, true, false)")
Dynamically evaluates an arbitrary formula using normal Excel-like formula syntax (not script)

recordCount() or recordCount(subset)
See http://www.visokio.com/kb/functions-guide#RECORDCOUNT

subset("Field name", "Field value") or subset("Field name", "Field value", "Operator")
See http://www.visokio.com/kb/functions-guide#SUBSET

subset_sum("Field name") or subset_sum("Field name", subset)
See http://www.visokio.com/kb/functions-guide#SUBSET_SUM
Note that all SUBSET_? functions are supported in JavaScript. The SUBSET_UNIQUESLIST function is slightly different in that subset_uniquesList() returns a JavaScript array rather than a single String containing a comma-separated list.

textvalue(value) or textvalue(value, pattern)
See http://www.visokio.com/kb/functions-guide#TEXTVALUE

More to come.

Example IOK file

Open the attached file in 2.7 b96 or later. This shows a table whose rows come and go as you filter. Try filtering by the Category field. Also try dragging the date sliders to see the numbers update and the rows disappear.

Expand the toolbars (or choose Toolbars > View toolbars), then click the "<>" icon to view source. The source is well-commented and shows different ways of working with scripting.

Formulae: Field/Cell direct access functions CELL, etc. (2.8+)

steve — Tue, 17 Jul 2012 09:56:49 -0400

These 4 new formula functions allow you to access arbitrary cells in the data. Also available in scripting.

CELL
FIELDCOUNT
FIELDNAME
FIELDNUMBER

Formula examples to obtain cell values

CELL([Price], 53)

CELL(FIELDNAME(3), CURRENTROW()-1)

Javascript preprocessing example to list all field values


for (n = 1; n<=fieldCount(); n++) {
  out.println(fieldName(n)+": "+cell(fieldName(n), 1)+"
");
}

DataManager: "Custom Script" Operation (2.8+)

steve — Tue, 17 Jul 2012 08:36:04 -0400

Omniscope 2.8 introduces a new operation called "Custom Script".

The "Custom Script" operation allows you to execute arbitrary Javascript to transform one or more inputs into a new output.

The result of the Javascript must be a two-dimensional array, indexed as [column][row] (i.e. [field][record]). You can optionally choose to provide the field name as the first row in this 'table'.

The inputs are accessible via "input1", "input2", etc. and also via "inputs[0]", "inputs[1]". These objects provide access to all regular Omniscope Javascript functions, such as:


input1.recordCount()
input[0].subset_sum("Price")

Here is an example script which simply ignores inputs, and produces static data:


[
  ["1, 1", "1, 2", "1, 3"],
  ["2, 1", "2, 2", "2, 3"],
  ["3, 1", "3, 2", "3, 3"]
]

Here is another which returns a 2x1 table of metadata about two inputs:


[
[
input1.recordCount()
],
[
input2.recordCount()
]
]

Here is another which does a sideways concatenation of multiple inputs:


working = new Array();
for (i in inputs) {
  var input = inputs[i].dataArray(null, null, true);
  for (n in input) working.push(input[n]);
}
working;

See attached example. Requires 2.8 or later.

Scripting in formulae (2.7 enhancements)

steve — Sun, 30 Oct 2011 11:22:37 -0400

Omniscope 2.6 introduced support for basic scripting in JavaScript within formulae using the SCRIPT function:


SCRIPT(`
  if (a < b) {
    "a is lower";
  } else if (b < a) {
    "b is lower";
  } else {
    "a equals b";
  }
`, "a", 5, "b", 7)

But this required you to pass any data-driven values to the script using a list of additional arguments to the SCRIPT function. This can be quite limiting.

New from 2.7 b100: data-driven scripting functions, as introduced for the new script pre-processing feature of the Content View, inside the SCRIPT formula function:


SCRIPT(`
  // Running total of Coupon field for all Categories up to the current Category:
  var array = subset_uniquesList("Category");
  var total = 0;
  var curval = refVal("Category");
  for (var i in array) {
    var val = array[i];
    total += subset_sum("Coupon", subset("Category", val));
    if (val==curval) break; // Stop here
  }
  // Result value:
  total;
`)

(Note also the use of back-quotes in formulae, making it easier to use regular quotes inside the script)

For a full list of supported functions, look for those with "In script" shown in the functions guide.

New website & video tutorials

steve — Sun, 20 Nov 2011 10:27:36 -0500

Visokio.com has been updated, rewritten and greatly simplified during recent weeks.

As part of this, instead of a user guide, we are providing a new set of video tutorials explaining everything about Omniscope to an intermediate level, with the Knowledge Base providing further information.

See: Video Tutorials

Feedback welcome.

Integration: ILF link files with parameter overrides (2.8+)

steve — Mon, 19 Mar 2012 12:51:04 -0400

A previous blog described how a 3rd party application can launch Omniscope to open a file with customised DataManager parameters:

http://forums.visokio.com/discussion/1376/new-command-line-arguments-for-datamanager-parameters/p1

We have now extended this concept to ILF link files.

(An ILF file is like a shortcut, which doesn't contain any real data. Instead it instructs Omniscope to open a file from a URL written inside the ILF file. You would use ILF files typically if you want to force Omniscope to open an IOK file directly from the IOK file's URL, rather than allow a web browser upon click to download the IOK file to a temporary folder first. This would allow you to use domain locking, for example.)

Use the new Server Edition XML action "Create Link File" to create ILF files which customise DataManager parameters. As with command-line parameters from the above linked post, you can specify the name of DataManager parameters and the value to use, which overrides the current value in the file. Using this method, you can override an unlimited number of parameters.

For example, the attached two ILF files instruct Omniscope to open the IOK file attachment from the above linked post.
- The first overrides the parameters to show the first 5 "abbey" tickers.
- The second overrides the parameters to show the first 3 "hbos" tickers.

SCP/SFTP and SSH port forwarding (2.8+)

steve — Wed, 30 May 2012 13:40:06 -0400

Omniscope 2.8 has today been updated to support SCP and SFTP for secure file copy to/from remote servers, and SSH tunnels with port forwarding for secure access to remote services such as databases.

Available shortly to alpha-partners.

SCP and SFTP

SCP and SFTP are secure file transfer protocols which use an SSH connection to transfer files to/from the SSH server's filesystem.

To use in Omniscope, from 2.8, enter the following URL syntax into the URL Data Source or Data Output block (for transferring in either direction, like an FTP URL):

scp://host:port/path/to/remote/file.csv
sftp://host:port/path/to/remote/file.csv

Be sure to tick "Requires authentication" and provide both a username and password.

SSH tunnel with port forwarding

An SSH tunnel with port forwarding allows you to "tunnel" access to private remote services (such as database servers) to your local Omniscope.

For example, you might use this to connect to your company's intranet database server from outside your company.

This is done by mapping a remote host & port accessible from the SSH server, to a local port, then updating your connection details to use the local port instead.

To configure, in any Omniscope 2.8 DataManager block which connects to a networked service, such as a database source or output:

Open the "Block options" menu (spanner/wrench icon)
Tick "Use SSH tunnel (advanced)"
In the dialog, configure the SSH session details (host/username/password), and port forwarding as required (see below)
Click OK

Once configured, the tunnel will be opened during any execution of the block, and closed immediately after. You cannot have two tunnels open simultaneously with the same local port, and if this is attempted, the Omniscope will process each conflicting block sequentially.

Port forwarding

With local port forwarding, a localhost port mirrors a remote service accessible from the SSH service. For example:

You are connecting to an intranet database from outside your organisation
The database server is internally called "BigDatabase" and is on port 3306
Only your SSH server is accessible externally
You would typically configure port forwarding as:
Local port: X
Remote host: BigDatabase
Remote port: 3306
You would then configure your database connection as:
Host: localhost
Port: X

(where X is an arbitrary free local port - it could be the same, 3306 in this case, or any number between 1025 and 65536, providing no service or other port forwarding is running locally on that port)

DataManager: "Field metadata" operation (2.8+)

steve — Mon, 02 Apr 2012 06:28:49 -0400

New in 2.8: the "Field metadata" operation.

This is a very simple operation currently without any configuration options. For every input field, it creates an output record describing that field, with attributes such as Name, Type, Min, Max, etc.

DataManager: More attributes in IOK "File metadata" block (2.8+)

steve — Wed, 11 Apr 2012 15:17:58 -0400

New in 2.8, the "File metadata" data source (available in DataManager and from "File > Open folder of files/media") now includes basic metadata from within IOK files:

- Record count
- Field count
- Field names, comma-separated
- Source file/URL/summaries, comma-separated
- Tab names, comma-separated (includes hidden tabs).

Layout: Formulas in page/view headers (2.8+)

steve — Wed, 18 Apr 2012 06:11:06 -0400

2.8, available shortly to alpha partners, now supports formulas in page and view headers. So, if you are using view headers (via view "Tools > Toolbars and styles > Header"), you can now insert formulas, which behave just as with formulas in page/view footers and in the Content view.

Formulas in headers currently don't have a wizard and have to be entered directly using {{double curly brackets}} when editing the header as follows:
This view has {{ RECORDCOUNT() }} records showing
or
Data for category "{{ SUBSET_SINGLETON([My Category Field]) }}"

Formulae: Fuzzy text matching-Soundex & Levenshtein (2.8+)

chris — Thu, 12 Apr 2012 11:52:43 -0400

Hi,

In Omniscope 2.8 we have added the ability to perform fuzzy matching in the Record/Field filter operations and in the Merge/Join block.

In both cases you can select either the Soundex or Levenshtein matching routines. Please let us know if there are any other routines you would like us to include.

In the Record/Field filter blocks you can perform fuzzy matching by selecting a fuzzy match routine from the operation drop-down. So whereas before you might have chosen "Filter records where Employee name = 'Angelina Joline'", you can now create the rule "Filter records where Employee name fuzzy matches 'Angelina Joline". Depending on the matching routine selected this may match records "Angelina Joyline", "Angelina "Joleen" and other similar names.

In the "Merge/Join" block you can specify the fuzzy match routine in the operation drop-down for each join criterion specified.

Please let us know if you have any questions.

DataManager: Database source field picker (2.8+)

chris — Wed, 11 Apr 2012 13:06:29 -0400

In Omniscope 2.8 we have now added a field picker to the Database configuration options. This allows you to select which fields to import without having to customise the SQL statement.

DataManager: "Batch append links" merging feed (2.8+)

chris — Tue, 03 Apr 2012 06:58:45 -0400

In Omniscope 2.8 we have added a new merging feed, "Batch append links". This feed requires one or more URL's as an input. It will then append the data obtained from each of these URL's into a single Omniscope tabular data set. It is very similar to the Batch Append operation, the difference being that in "Batch Append" the list of URL's are generated dynamically based on a selected directory.

Formulae: ROUND function (2.8+)

antonio — Mon, 02 Apr 2012 11:59:52 -0400

Available from 2.8, the ROUND function is equivalent to that in Excel, i.e. rounds a number to a specified number of digits.

Syntax:
ROUND(number, num_digits)
where:
- 'number' is the number that you want to round.
- 'num_digits' is the number of digits to which you want to round the number argument. Default 0 if not specified.

Samples:
ROUND(123.7825, 2) = 123.78
ROUND(123.456, 1) = 123.5
ROUND(123.456, -2) = 100)

Version 2.7: now the Stable Release

steve — Thu, 16 Feb 2012 04:20:18 -0500

Omniscope 2.7 has been in development since August 2011, during which time it has had extensive use by our alpha partners and beta testers. We've now finished all functionality and fixed all major known issues, and have established the latest build as the stable release. Please check the download page for the latest build.

Download:
http://www.visokio.com/download

Please continue to report any errors from within Omniscope, and any feedback on these forums.

Outputs: Custom Executable JAR (2.6)

steve — Fri, 30 Sep 2011 12:50:58 -0400

From Omniscope 2.6, the Custom Executable Jar utility was added to Omniscope, available for users of the Desktop or Server/ServerPlus licenses.

This deployment option allows you to create a standalone free Viewer JAR file with bundled data file, and optionally your own branding in the Viewer (licensing permitting). You can host this JAR file on any web server and provide a link to it. Users will click the link to download it, then open/run the downloaded file. Omniscope will launch and open the bundled file automatically, avoiding any authentication issues which may arise with Web Start/JNLP.

Instructions:

If you wish to re-brand, ensure you have a ServerPlus license and a recent build of 2.6+
Choose Settings > Advanced > Miscellaneous > Create custom JAR
Specify the full path to the IOK file
Optionally, specify the full path to the JAR file. Leave blank for the latest version.
Optionally, provide re-branding customisations. See http://www.visokio.com/kb/rebranding
Specify the full path to the output directory, e.g. your C:\Users\YourName\Desktop
Click OK
Wait for the progress dialogue to complete
Test by double-clicking the JAR. Omniscope Viewer should launch and display the bundled IOK file.

The deployed free Viewer will be identical to Web Start in terms of functionality and you may not be able to rely on whether the operating system chooses to launch Java in 32-bit or 64-bit mode, so be sure to test on your target platforms, especially for larger data files.

Please post back any feedback here.

Minor scheduler enhancements (2.7 b213+)

steve — Tue, 24 Jan 2012 09:18:55 -0500

The scheduler task allows you to specify a time-out in seconds, but this only takes effect if the "fork" global config option is enabled. This has now been clarified in the interface.

In addition, "Chain" actions now have two new options:

Time-out (seconds): If non-zero, allows you to put a limit on the entire sequence of actions. If the limit is reached, Omniscope will attempt to terminate the action as a failure. This works regardless of the "fork" setting.
Propagate failure: If unticked, failures (including timeouts) are isolated to the chain action. Any enclosing chain actions will continue unaffected. The log will show the failure details.

Using these options and a hierarchy of nested chain actions, you can create any effect you want in terms of time-outs and failure handling.

Please report any issues with this as errors in Omniscope or by commenting below.

Copy/Paste settings

mustafa — Thu, 12 Jan 2012 06:45:56 -0500

From 2.7 b193 you will be able to copy/paste the following settings within the same file, or between files.

Within the same file

Aggregation settings

You can copy/paste aggregation settings of a particular view by clicking the Aggregation dropdown of a particular view and then choose “Tools” button above the preview table.
Tab settings

You can copy/paste a tab in several ways from the Tabs menu in the main toolbar or by right-clicking the tab.

The tab settings once copied can be applied in 2 different ways:
- Replace the entire tab with another tab’s settings. This can be achieved by selecting paste
  in the appropriate tab or by right-clicking on a different tab and choosing paste.
  
  Note: You don’t need the tab to be active in order to apply the setting. You can simply paste the settings by right-clicking the target tab.
- Paste the tab as new tab by choosing “Paste from clipboard” which is right next to other create tabs options (Plus ‘+’ button in the tab bar or from Tab > New).
View settings

You can copy/paste view settings from “Tools” menu from the View’s toolbar. Copying view settings will copy aggregation as well as data subset settings among others.

If you try to paste a settings from different view type you will be prompted to allow you to confirm switching the view type. You can also add the copied view as a new view from the “Add view” button.

Between files

You can apply all the above settings between files by copying settings from one file and then choosing paste option in the other file.

You can also import tab settings from another file by choosing “Import tabs from another file…” which is located in the same place “Paste from clipboard”. Choosing this option will prompt you to choose a file and allow you to pick which tab settings to import.

If the settings are not compatible, for example, if you are copy/pasting across files and some of the fields do not exist then settings which refer to invalid field will be ignored and not applied while other settings will be applied.

Web/News/Image/Video search (2.7)

chris — Fri, 23 Dec 2011 11:10:26 -0500

Hi,

In the latest version of Omniscope 2.7 we have removed the Yahoo web/news/image search.

These have been replaced with generic Web/News/Image and Video search blocks. When you add one of these blocks you need to select the search service (for example Yahoo or BING). After selecting the appropriate service the configuration options for that service will be displayed.

In the future we plan to add more search blocks (blog search, forum search etc). We also intend to add additional search services, such as YouTube. Please let us know of any other services you think might be useful.

Have a good Christmas!

Function filtering (2.7)

steve — Mon, 05 Dec 2011 06:29:26 -0500

New in 2.7: filtering in all column function menus, by "Common" or "All". Default "Common", meaning only commonly-used functions are revealed (sum, mean, etc.), with advanced choices like "Unique value count" hidden until the filter is toggled.

Custom Search filter device (2.7 b138+)

mustafa — Wed, 30 Nov 2011 10:52:58 -0500

Starting from 2.7 b138, you will be able to add a new filter to the sidebar called "Custom Search".

Custom Search filter device

This filter will behave much like the "Search All" filter, but it will allow you to choose which fields to search rather than all the fields.

You can choose which fields to search from the "Fields to search" menu, accessible from the filter device tools menu (by clicking the down pointing triangle).

Note: This filter will return rows that contain the search text in any of the selected fields. If you want the rows containing the search text in all the selected fields, you will have to use per-column filters.

Text variable (2.7 b136+)

steve — Mon, 28 Nov 2011 12:11:25 -0500

Introduced for 2.7 b136: A new variable type "Text" to complement the "Text choice" variable type.

"Text" variables show a text field and allow you to enter arbitrary text, allowing you to construct custom filter devices using formulae and frozen devices, such as a filter over 2 fields, as discussed here: http://forums.visokio.com/discussion/1113/search-though-fields

Miscellaneous UI enhancements

mustafa — Mon, 21 Nov 2011 13:01:52 -0500

Here is a summary of some new UI enhancements we have introduced in Omniscope 2.7 (b127+).

Enhancements to field pickers

You now have the ability when you click the "Spanner" icon in appropriate field pickers to:

Hide all fields from showing in the drop-down by clicking "Hide all".
Apply settings on per-field or globally by clicking "Apply to..."
- all drop-down menus in this tab
  
  Applies the drop-down configuration to all drop-downs in the current tab.
- this drop-down menu in all tabs
  
  Applies only this type of drop-down in all tabs. You would use this option if you have would like all certain type of menus to look the same. For example, you may customised your "Colour by" drop-down to have only a certain fields, and you
  want to only apply the configuration to all "Colour by" drop-downs in all tabs without affecting other drop-downs.
- all drop-down menus in all tabs
  
  Applies the configuration to all drop-downs in all tabs.

Note: If you wish to apply per-field configuration only then you should use the "Apply to..." next to the "Show" button, rather than the global "Apply to..." at the top of the toolbar.

Toolbars

Ability to control the transparency of the View toolbars from "Style > Application styles > Edit..." and from "Toolbars > Toolbar settings > Toolbar transparency".
Ability to hide the main menus (File, Data, Tab, Style, Toolbars, Settings and Help) by right-clicking on the main toolbar.

You can now also control what is shown/hidden from the main toolbar through configuration of individual styles presets.

Hide table view headers

You can hide table view column headers by reducing the height of column height to zero

Other

Add view menu now always available from Style menu.
Easy access to ability to hide/show formula field blue text colour from the Formulas menu.
Simplified how you can set Cycling and Tab bar to show, show on hover or hide.
Ability to permanently hide the view barometer from the "Toolbars and styles" menu from Tools drop-down from each view's view toolbars (available from 2.7 b128+).

Publishing to Excel template (2.7+)

chris — Wed, 16 Nov 2011 12:38:33 -0500

We have added a new feature in the Excel output block that allows you to specify what action you want to take if you are publishing to an Excel worksheet that already exists.

You can choose from:

- Replace worksheet. This will delete the worksheet and replace it with a new worksheet every time you publish the data.

- Replace data, keep formatting. This will not delete the worksheet, but will instead delete all of the data inside the worksheet and replace it with the data you're publishing.

- Overwrite data, keep formatting. This will not delete the worksheet or the data, but will instead overwrite the data with the data your publishing. Any data outside of the published data range will be retained.

We have also added the ability to specify whether you want to write the headers and the position of the data (the top-left column/row).

The Replace data/overwrite data options allow you to create an Excel "template" file that contains the formatting (colours, fonts etc.) for the data you want to publish. When you publish to this worksheet the formatting will be retained after your data has been inserted.

Filter by name/type in field Organiser (2.7)

chris — Wed, 16 Nov 2011 12:31:16 -0500

We have now implemented the ability for you to filter by name or field type (numbers, text, dates) in the field organiser operation. These options are available at the top of the configuration panel, above the list of fields.

This functionality is particularly useful if you are working with data-set's that contain a large number of fields, allowing you to navigate to the relevant field without having to find it using scrolling.