Sources

Omniscope allows you to import data from a wide variety of data sources. Each source allows you to fine-tune how the data will be imported.

File

The File source lets you import data from a local file.

Options

File selection

The file selection section allows you to choose what file you want to import. Click on the "Browse" button to select a file.

Omniscope supports the following file types:

Visokio Omniscope file formats (IOK, IOM, IMF).
Comma Separated Values (CSV).
Tab Separated Values (TSV).
Plain text files (TXT).
Excel workbooks (XLS, XLSX).
Outlook (PST).
Quattro Pro Spreadseet (QPW).
Web access logs (LOG).
XML data files (XML).

After you have selected a file the options for configuring how you want to import the file will be displayed. These options vary depending on the type of file you have selected.

Omniscope file options

The following options are available when you select a Visokio Omniscope file type (IOK, IOM, IMF):

File password. Enter the password used to open the file. Leave blank if the file has no password.
Fields. Select which fields you want to import.
Formula recalculation. If the selected file has formulas fields this option determines how the data in these fields is recalculated. Select "None" to disable formula recalculation altogether. Select "Formulas that change over time" to only recalculate formulas with values that can change depending on when the formula was calculated. Select "All" to recalculate all formula values.
Data refresh. If the selected file was obtained from an external data source this option determines whether the data is refreshed. Select "Disabled" to disable data refresh altogether. Select "Automatic (if detectable)" to refresh only if the data source has changed. Select "Forced refresh" to refresh whenever the data is imported.

Text based file options

The following options are available when you select a text based file type (CSV, TSV or TXT):

Type. Allows you to select the structure of the data inside the file. Select "Text, delimited" if each field in the data is delimited by a specific cell separator character. For CSV files the default separator is the "," character and for TSV files the default separator is the TAB character. Select "Text, fixed width" if each column of the data is specified a set number of characters. If you have a data in a text file where the first column has exactly 10 characters, and the second column has exactly 8 characters and so on, then you should select this option. Select "Text, XML" if the data inside the file is XML. Not all XML files use the "XML" file extension.
Select table data. Click this button to identify the header rows, data rows and columns inside the selected file. For more information see the Table data selection help page. PENDING.
Has header. Select this option if the file has one or more header rows.
Header rows. Enter the first and last header rows.
Data rows. Enter the first and last data rows. Leave the last row blank if you want to read all rows.
Skip empty data rows. Select this option if you want to skip all data rows with no data.
Auto-recognise. This option allows you to configure whether Omniscope will attempt to recognise numbers and dates. Leave these options unchecked to import the data as text.
Auto-convert to empty. If this option is selected the following will be converted to empty values (nulls): "null", "None given", "Unknown", "Not known", "N/A".
Cell separator. "Text, delimited" type only. Enter the field separator character.
Cell quote character. "Text, delimited" type only. PENDING.
Encoding. Select the encoding of the file. If you see any unusual characters in the data you can try changing this to UTF-8 or another encoding.
Locale. PENDING.
Missing field prefix. Enter the field name prefix used when a missing field is encountered. If your data contains 3 missing field names and you select the prefix "Field" the missing fields will be named "Field 1","Field 2" and "Field 3".
Break points. "Text, fixed width" type only. Click this button to select the break points in the data. The breakpoints define the number of number of characters in each column. For more information see the Set break points help page. PENDING

Excel file options.

The following options are available when you select an Excel file type (XLS, XLSX).

Password. Enter the password used to open the file. Leave blank if the file has no password.
Import. Select "Single worksheet" to import one worksheet only. You can choose which worksheet you want to import from the dropdown below. Select "Appended worksheets" to select one or more worksheets to append into a single data-set. You can choose which worksheets you want to append in the dropdown below.
Reader. This option allows you to select how you want Omniscope to read the Excel data. Your choice should depend on whether you have Excel installed, how much data you are trying to read and the content of the data. If you are experiencing problems with one reader you can try switching to a different reader. The "Excel as CSV" reader will open the file in Excel and the force Excel to save the data as a CSV file before importing it into Omniscope. This reader is normally the best method for reading the data as it uses Excel to interpret any formulas inside the workbook. You must have Excel installed to use this reader. If this option is selected and you don't have Excel installed the "Omniscope" reader will be used instead. The "Excel direct" reader will open the file in Excel and then communicate directly with Excel to obtain the data. You must have Excel installed to use this reader. If this option is selected and you don't have Excel installed the "Omniscope" reader will be used instead. The "Omniscope" reader is the only reader that can be used if Excel is not installed. It uses Omniscope to decode the file format. This reader requires more memory and may not work with certain formulas.
Table data, Has header, Header rows, Data rows, Skip empty data rows, Auto recognise, Auto convert to empty. For more information on these options please refer to the documentation in the Text based file options section of this page.

URL

The URL source allows you to import data from a remote file.

Options

File selection

The File selection section allows you to select the URL you want to import. You should enter the web address in the "Address" text field. Examples of valid addresses include:

http://www.mysite.com/example.iok [1]
ftp://ftp.mysite.com/public/example.txt

Select the "Requires authentication" option if the server requires a username and password in order to access the data.

The URL source supports all of the file types available in the File source. After selecting a file the configuration options for the selected file type will be shown. For more information on these options please see the File source help page. PENDING.

Database table

The Database table source allows you to connect to and import data from a relational database.

The connection details section of the panel allows you to select and connect to a Database. Omniscope supports a wide variety of different Databases. If your database isn't on the list then you should select ODBC or JDBC (Advanced).

After selecting a Database you need to configure the information required to connect to that database. Different databases require different sets of information. For example, when connecting to Microsoft SQL Server you should select a driver and the host, port, host (optional), database name (optional) and authentication information.

After entering the required connection information click on the 'Connect' button. If there are any problems connecting to the database an error message will be displayed. Successfully connecting to the database a list of the tables available to import will be displayed in the Table selection dropdown.

Advanced users can choose to import data using a custom SQL statement rather than by simply selecting a table.

List of values

The List of values data source allows you to enter one or more values that will be converted into a single-column Omniscope dataset. You can then use this as an input into any operation or feed in the DataManager workspace.

The List of values source is particularly useful as an input to a feed, such as the Twitter search or Yahoo search, where you want to supply a pre-defined set of search values.

Options

Field name. Enter the name of the field containing the list of values that you want to supply.
Values. Enter the values. Each value will appear as a separate row in the data.
Value separator. Enter the separator for each value. The default separator is the newline character (represented using "\n"). If the newline separator is used each value should be entered on a separate line. You can only define one separator character.

Sequence of values

The Sequence of values source generates a single-column Omniscope dataset containing a discrete sequence of values between a start and end value. You can use this source as an input into any operation or data feed.

Options

Type. Allows you to determine the type of sequence generated. You can choose to generate a number sequence or a date sequence.
Start. Defines the first value in your sequence.
End. Defines the last value in your sequence.
Increment. Determines the difference between each value in your sequence.
Field name. The name of the field in the resulting dataset that contains your value sequence.

File metadata

The File metadata source generates an Omniscope dataset that contains file metadata related to a subset of files in a specific directory. Each row in the dataset represents a different file.

Some examples of file metadata include:

The name of the file.
The size of the file in bytes.
The last modified date.
(Image files only) The size of the image.

Options

Folder. Specifies the directory that contains the files you want to scan. You can choose to include the files in all subfolders by selecting the Include subfolders option.
File type. Allows you to specify the types of files you want to scan. You can choose to scan All files, Images or a Custom list. If you choose the Custom option you can specify the filenames of the filenames you want to include. So, for example, if you want to scan all JPG and TXT files you should enter '*.JPG,*.TXT' in the filename filter.
Tags in filenames.

Batch append files

The Batch append files data source allows you to append the contents of a set of data files within a designated folder.

Options

The File selection section allows you to configure which files you want to append.

File type. Allows you to select the types of files that you want to import. All file types supported by Omniscope are listed here. When you select a particular file type, the configuration options for that file type are displayed, so if you choose the 'Microsoft Excel file type' then the configuration options specified (worksheet name, header/data rows etc) will be applied to all workbooks matching the selection criteria. If you need to import files requiring different import configurations you should consider using mulitple Batch append sources or importing the files individually. Most of the options available in the File and URL sources are available in the Batch append source.
Folder. Allows you to select the directory containing the files that you want to append. At the present time the Batch append source does not allow you to select FTP directories.
Scan sub-folders. Determines whether files in sub-folders inside the selected directory will be scanned. If unchecked only files in the selected directory will be appended. If checked files in the selected folder and all child folders will be appended.
All/part of filenames. Allows you to filter which files you want to append. Whenever you change the file type the filename will change to reflect your selection, so if you choose the 'Excel workbook (Excel 2007)' file type the filename will automatically change to '*.xlsx'. You can enter multiple filters by separating each one using the ',' character. So for example if you want to append all .TXT and .DAT files you would enter '*.TXT,*.DAT'.

The Source field section determines whether Omniscope will create an additional field in the dataset that contains the file each row of data was obtained from. You can enter the name of the file and choose whether to remove the file extension. This feature is particularly useful if you are appending a series of files containing timestamps inside their filenames.

Lookup data

The Lookup data source allows you to pick an Omniscope lookup dataset. These datasets provide useful information that can be combined with your existing data sources to provide additional data. Examples of lookup datasets include postcode and country location data.

Demo data

The Demo data source allows you to pick a demo dataset. You can use the demo data to test the functionality of the operation, feed and publisher blocks.

OLAP cubes

OLAP cube.

The OLAP cube data source provides connectivity to MDX-compliant multi-dimensional data secttructures (cubes) such as Microsoft SSAS and other cubes.

Omniscope DataManager workspace currently supports direct connections to Microsoft Analysis Services and some other MDX-compatible mutli-dimensional cube structures from other vendors.

In order to use the OLAP connector you will need to configure HTTP access to the MDX server. Most OLAP sources support this, however you may need to configure access on the server.

The following link [2]provides details on configuring HTTP access to SQL Server 2008 Analysis services. Information on configuring access to other cube providers can normally be found on the internet or by speaking to your system administrator.

Visokio made the decision to provide access cubes via MDX because it provided a generic solution, allowing connection to several different providers. The downside of this solution is that it does require some pre-configuration and data retrieval may be slower than a direct connector. Visokio intend to implement direct connectors for OLAP servers such as Microsoft SSAS in the future. This development will be dictated by demand from clients.

Options

The connection details section allow you to specify how to connect to the OLAP server.

Server URL. Enter the name or IP address of the OLAP server.
Username/Password. Enter the authentication details that you use to connect to the OLAP server. Once you have entered the connection details click the "Connect" button. If the connection was successful a list of OLAP cubes, dimensions and measures will be downloaded and available for selection.

The cube data section specifies what data you want to retrieve from the OLAP server. You can only select this data once you have successfully connected to the OLAP server.

Cube. Select the Cube that you want to import.
Dimensions. Choose the dimensions available in the selected cube.
Measures. Choose the measures from the selected cube.
Check max rows. Select this option if you want Omniscope to check that the maximum number of rows obtained from the dimensions selected does not exceed a specified value. This check is done before the data is imported into Omniscope.
Custom MDX. Select this option if you want to modify the MDX or enter a custom MDX statement. Not all MDX will be compatible with Omniscope.

Formatting

The formatting secion determines how the data will be structured when it is imported into Omniscope.

Pivot data. Select this option if you want to pivot the data before importing it into Omniscope.