Tokenised Data

Using multiple values within each cell

Sometimes, more than one value is associated with certain attributes of a record. For example, shoes can be available in more than one size, or you may have more than one image associated with with a house (or whatever). Shoe sizes and multiple images of a house are examples of many-to-one relationships. Omniscope handles these kinds of relationships using a 'Tokenised' data subtype for Text and Category fields (columns).

When you declare a Text or Category data type to be 'Tokenised,' it means each cell in a field can contain more than one value, i.e. multiple "tokens" separated by a character you decide. The default separator is a comma, but you can choose to use any character: a space to separate words, or an unusual character such as a  " \ " backslash if you need to use commas within your values.

The example below (you can download a copy of this example file here) is a sample shoe store data set.  Each record (row) corresponds to a pair of shoes, and the 'Sizes available' field contains size numbers in stock (38, 39, 40, 41 etc.). The 'Sizes available' field has been typed a Category field since it contains only a few discrete values, the size numbers must be individually filterable (show me only my size) and the numbers treated as text, not numbers used for calculations.

Example: using Data > Manage fields > Configure > Field options, we can set the data sub-type of the 'Sizes available' Category field to be 'tokenised category' with the default comma separator dividing the values within the cells.  We then check the data to ensure that each available shoe size is entered in the 'Size' column with a comma separator as shown in the Table View above. We can quickly check that all values have been entered correctly in the Sidebar query device for 'Size', because each size should appear separately.

In the example below, the user has selected a specific shoe style. Notice that in the Chart View, the highlighting for a specific pair of shoes highlights the full range of sizes available. This behaviour is the same in all other views that display breakdowns of Tokenised Categories, such as the Pie and Tile Views.

If there were multiple images for each pair of shoes, we would have done the same thing in the 'Photos'  image references field (column), so that we could relate more than one image to each pair of shoes, and display the multiple images in the details display.