De-query Option

Using the De-query Option

Returning records to the filtered-in data subset selectively

The de-query option allows you to modify data subset definitions selectively to return records already filtered OUT to the filtered IN data set.

Data Universes (2.3)

Data Universes

MOVE Data OUT, KEEP it IN, and Using the Basket

Relates only to version 2.3 and earlier. Data Universes have been reclassified as Data Subsets selectable using the Data Subset drop-down selector in each individual View Toolbar,  and configured using the Queries drop-down on  the Main Toolbar (2.4)

When Omniscope filters records, where do the excluded records go? Don't worry...they are not deleted! Based on your filtering decisions, Omniscope classifies each record (row) in a given file as being in one or more Data Universes:

• Green 'IN' Universe- records still under consideration
• Red 'OUT' Universe- records moved out, excluded from further consideration
• Black 'ALL' Universe- all records in the file/data set, regardless of queries/filters set
• Gold 'BASKET' Universe- records saved to the basket during the session
• Blue 'SELECTION' Universe- records selected at any given time

All of these Universes can be viewed at any time in any of the 12 available views in Omniscope. Click on  Add View on the Main Toolbar, then select the Show other data universes option at lower right.

 

Tick the box representing any combination of Views and Universes to see the records (rows) currently in each universe in each view.

If you save an Omniscope file, the contents of the IN and OUT Universes will be preserved, and the file will open with the barometer showing the same IN/OUT split as when the file was last closed. However, the contents of the BASKET and SELECTION Universes will be re-set. Always remember to do something with the contents of your  BASKET universe before you close the file, because this selection of records will not be there when you open the file again. 

Let's explore the use of Data Universes in Omniscope, while re-visiting the filtering aspects of navigating .IOK files. Recall that there are 2 filtering approaches in Omniscope, which can be used interchangeably at any time, assuming at least one SideBar with instant filter/query devices is showing.:

1. Sidebar filter/query devices (known as instant query mode)
2. Select > Move > Keep sequences (known as power query mode)

In the Tutorial below, we will see in more detail the difference between single-step, 'instant' filtering using Sidebar filters/query devices, and multi-attribute, potentially multi-layered 'power' queries using Select > Move > Keep sequences. If you are viewing both the IN and the OUT universe, it is also possible to do dual-directional, multi-layer power queries...which are difficult, and sometimes even impossible to do programatically!

Using Data Universes

For this tutorial, please download another, especially-configured version of the Everton Football Universes [1] file. This version of the Everton file has been configured to open with five Table Views, one for each Universe, and with only one Side Bar open:

 

The View icons on the far left of each View Toolbar are coloured to show which of the 5 Universes is being displayed in the view. In this demo, the upper left-hand Table View is showing the green IN Universe, all 23 players. The upper right-hand Table View is showing the red OUT Universe, currently empty because no players have been filtered out. The full-width black ALL Universe Table View in the middle will always show all players, regardless of filtering. The gold BASKET Universe Table View is showing 'Basket Empty', since no players have yet been placed in the Basket. Finally, the bottom blue SELECTION Table View is empty, indicating that no records are yet selected.

Using the Side Bar, type the letter 'B' into the top text query device labelled Surname. Notice how the right-hand green IN Table View changes to display only the 3 players whose surnames begin with B, and the left-hand red OUT Table View now displays the records for the 20 players whose surname does not begin with 'B'. A glance at the barometer confirms that the green IN Universe contains 3 players whose surnames begin with B, and the red OUT Universe contains 20 players whose surnames all begin with other letters. In the Side Bar, back-space over the letter 'B' to clear the Surname query device, which will Reset the file, since this was the only extant query/filter setting.

Notice that when using the Sidebar devices, the bottom Table View showing the SELECTION Universe did not show any records being selected, and that moving 20 players to the OUT Universe was instant. 

Now let's try using Selection, which can be very useful for drilling-down and confirming details before deciding whether to execute a Move or Keep operation. 

Go to the upper right green Table View showing the IN Universe, and click on the Position category Defender, which will display an animated highlight in blue. This means you have selected all the players who are Defenders, and their records will now appear in the bottom blue SELECTION Table View. A glance at the barometer will confirm that you have selected 9 Defenders out of the 23 on the team.

Let's say you want to take a closer look at the Defenders. On the main/top Omniscope Toolbar, click on the green check-marked icon labelled Keep. This will keep only Defenders in the green IN Universe and move all others to the red OUT Table View to the right. Notice that the SELECTION Table View has been cleared, pending you next selection.

Let's say you want to put all the English Defenders on the transfer market (don't ask...). In the Sidebar, using the 3rd device labelled Country, click on the category name England. Notice that all other nationalities are X-ed out and all non-English Defenders are moved to the red OUT universe, leaving only the 4 English Defenders remaining in the IN Universe. Now go to the green IN Table View Toolbar, and find the Add to Basket icon right-most on the toolbar. Click the Add to Basket icon, and notice that the gold Table View showing the BASKET Universe now shows the 4 English Defenders.

Click Reset on the Omniscope Main Toolbar. This will return all players to the IN Universe, and empty the OUT universe, but leaves the 4 English defenders in the Basket, as can be seen also on the top/main Omniscope Toolbar Basket icon to the right of the barometer.

Let's save the youngest English Defender for the future. In the gold Table View showing the 4 English Defenders currently in the BASKET Universe, click on the column header Date of Birth. This will set a primary sort (see the orange downward arrow appear in the column header) sorting the players in the basket from youngest to oldest. Double-click on the row header of the youngest player, Daniel Fox. This will display the details window pop-up for Daniel Fox. Now click Remove from Basket to remove Daniel's record from the basket.

To clear the BASKET Universe completely, click on the right-most Clear basket icon in the gold Table View Toolbar, or use the Omniscope Main Toolbar Basket > Clear basket command on the drop-down menu. More on Using the Basket... [2]

 

Text Filtering & Matching

Text Filtering & Matching Options

Filtering and matching text strings

 

Each text filter device has a drop down options menu that allows you to further refine the status and behaviour of the device.

 

 

Choosing the Show Text tools option provides more options.

 

The leftmost drop-down allows you to specify the text filtering options to be applied in this filter:

Substring (default) -

Word prefix -

Exact -

Regular Expression -

 

http://www.regular-expressions.info/ [11] 

Sidebars (2.4)

Using the Sidebars (2.4)

Instant filtering query devices and tools

Applies to version 2.4 only

The Sidebars are used to display filter/query devices and other options on either the right or left-hand side (or both) depending on the tab-specific settings on the Main Toolbar Controls drop-down menu:

Sidebar Display

Whenever either Sidebar opens, there are three menus displayed at the top; the Devices drop-down menu, the Tools drop-down menu, and the right corner [X] to close the Sidebar. The first time the Sidebar opens, three configurable display options (Reports, Search All, and Exclusions) plus filter devices for all fields (columns) in the data set will be ticked and therefore visible on the Sidebar.

To configure or re-configure the Sidebar, click on the Devices drop-down menu on the upper left:

Except in a general 'Data Set' tab intended to show everything, you will probably not want filter devices for ALL the fields (columns) to be displayed on most pages. Some field values may not be relevant to the message or useful for selecting/filtering the displays on a given page. Use the Devices drop-down menu to untick those fields to hide their filter devices. Use the drag hands at the right to move filter devices upwards or downwards to achieve the desired display on the Sidebar. You can also move the filter devices themselves into a new order by dragging and dropping them with the mouse up-wards or down-wards on the Sidebar.

The three configuration options at the top of the Devices list (Reports, Search All and Exclusions ) will be displayed on the Sidebar only if you leave them ticked. Otherwise, untick these boxes and Reports tabs will display across the top of the Main Toolbar (the default), Exclusions (Moves and Keeps with history) will show on the Main Toolbar (without history), and Search All will be available in the Portal View [5], when open.

Note: Sidebar settings are specific to each Reports Page, and must be changed page-by-page inside Reports Mode. Sidebar settings established outside Reports Mode (always hit Escape to make sure you are not working in a specific Report tab) will apply outside Reports Mode. If you create one or more Report Pages, you can either keep your existing Sidebar settings the same or change them to suit each Report tab.

Using Sidebar features

If you leave all three display configuration options ticked, the top of your Sidebar will look something like this:

Displaying Exclusions on the Sidebar has the advantage that you can expand the Exclusions device and monitor the history of your Moves and Keeps (see image below). This is especially useful when you are recording a power query sequence of Moves and Keeps to send to someone else as a query file for use with their copy of the file (see below for more on query files). Displaying Reports on the Sidebar is useful when there are very few, or when there are many Report Pages that need to be arranged in groupings. Like Search All above, any Sidebar text filter can be expanded to reveal a virtual keyboard for text search entry, with ANY and ALL multi-term search options. Omniscope text search works like a search engine, on one, several or all fields (columns)simultaneously. In any text filter, if you enter a single text string with no spaces, Omniscope will perform a prefix match, highlighting all records with text beginning with the search string. If you enter two text strings separated by a space and do not enclose them in "double quotes", you can search a field (column) of the data file for either one or both of the strings using the ANY or ALL search options. You can expand your search criteria to include values in other columns by simultaneously entering text strings in more that one filter device. To the right of each device you will see the hits in that column and at the top, in the barometer will show the combined record count.

 

Note: Omniscope does not yet support an EXACT match option, so searching for numbers and text can return records whose values are supersets of the search term, i.e. a search for '22 ' will also display 222, 223, etc. leaving you to select the final records manually. (support for EXACT was added in version 2.5)

Closing the Sidebar

Notice that whenever a query device is actively filtering the data set, the device panel colour turns orange:

Closing the Sidebar with filters set will prompt a message either to reset all the filters, or to convert the query settings to Exclusions. If you wish to keep the same record set in the IN Universe with the Sidebar closed, you must choose Convert to Exclusions, which means that the filter settings will be translated to the equivalent Moves and Keeps that you can manage directly in the open views without the Sidebar.

Sidebar Tools Menu

The Sidebar Tools drop down menu displays commands that affect the data and/or all other Sidebar query devices. The first three of these are especially useful when managing the creation of Report Pages and using them in live presentations.

Reset variables- Variables are input values not contained in a column of the data set that you can include in formula fields to facilitate dynamic modelling and sensitivity analysis. When you specify a Variable, you define a minimum, maximum and a default value in the range. Each variable defined in a file can display a Sidebar slider enabling users to change the assumed value. Clicking on Reset variables returns all variables to their default values. For more on defining and using variables, see Formulas & Variables [7]. Reset filters (query)- Clears all filter settings and returns all records to the IN universe...does the same as the Reset on the Main Toolbar. Reset configuration (views & layout) - returns the current view to a 'pristine' state, with all views closed and the Sidebar reset to opening default settings. Usually used to recover from confusing combinations of views and layout when constructing Report Pages. Clear colour-by - each device has a small colour-by icon in the upper left. Clicking the device icon imposes the colouring scheme for that field (column) on all the open views that use colouring, like the Table View, the Tree View, etc. This command removes all overriding colouring schemes that have been set in the Sidebar (but not those already specified in the individual View Toolbars) Halo function: Also include records with the same value in:  - a powerful feature for managing groupings or clusters within a data set. Setting the Halo function changes filtering behaviour to include/maintain all records sharing a common value in the specified field (column). For more information, see Using the Halo function. [8]

[7]

Expand all -  expands all visible Side Bar devices

Collapse all - returns all Side Bar devices to their collapsed state (default on opening)

Add new variable - brings up the Add Variable wizard...see Formulas & Variables [7]

Edit variables - brings up the Edit Variable wizard...see Formulas & Variables [7] 

Convert all to exclusions - Converts the current settings of all query/filter devices into Exclusions (the results of Moves and Keeps), thereby setting the IN universe independently of Sidebar settings...allows Sidebar devices to be reset and/or hidden without changing the current composition of the data subsets.

Import XML query file - XML query files exported from Omniscope by one user, say a business user identifying data quality issues with specific records (e.g. corrections or suspicious data) can be imported by another Omniscope user, say the Database Administrator, who wants to view just the records at issue and decide whether to parse the XML file to import the corrections into the source database.

Export XML query file - Exports the current query settings as an XML text file which other Omniscope users can open to recreate the same display, assuming they have the same .IOK file version open.

Right-click device menus

Right-clicking on any device displays a series of options for resetting/hiding the device, and changing the filter type. Changing the filter type is sometimes necessary to facilitate user interaction. In the example shown below, the field 'Node' is an integer number field, displaying as a range slider (the default) and with the 'Include empty (null) values' box ticked. If it was better to have users search for a specific number by typing it in, we could change the filter type to Text search...without changing the data type to Text.

 

 

Below the Filter Type command are a series of commands for changing the filter types for each field. These commands are the same as those accessible from the Main Toolbar: Edit > Manage Fields menu, and are discussed here [10].