The following example assumes that you already have an ArcGIS Online account with at least one feature layer configured. You can sign up for an account by clicking on "Esri account help" in the DataManager block noted below and following the link.
INSTRUCTIONS
- Create a new file in Omniscope and go to DataManager.
- Under "Feeds", drag on a new 'ArcGIS feature layers' block.
- Enter your ArcGIS Online username and password then click connect
- Configure the search criteria then click 'Search' to find relevant Feature Services.
- Choose one or more Feature Services then select the layers you would like to import into Omniscope.
- Load the data into Omniscope then drag a Map View into Omniscope to see the postcode areas.
UK specific
- Ordnance Survey Open Data BoundaryLine administrative low level (high quality): https://www.ordnancesurvey.co.uk/business-and-government/products/boundary-line.html
- Edina for admin/political/nhs boundaries: https://census.edina.ac.uk/bds.html
Worldwide
- Openstreetmap: http://wiki.openstreetmap.org/wiki/Shapefiles#Download_shapefiles
ArcGIS:
http://forums.visokio.com/discussion/2637
The ArcGIS API will have some of those boundaries that can be accessed with a free public account you just have to sign up for a trial to get some login information for the software. Also every year ArcGIS provides a couple of online MOOCs where you can get some extra help regarding their products.
To get these into Omniscope you need to
1) Convert OSGB shapefiles into WGS version shapefile/kml - import/load up your shapefile into your GIS package of choice and set the current projection to OSGB, then go into the save as/export options and change the resulting output into WGS projection.
I would suggest you save as .kmls as they appear to be the leanest option.
2) Generalise/clip the shapefiles to reduce the load time (also reduces accuracy), load up the shapefile in you GIS package, find the generalisation options either inbuilt in some packages i.e. model builder with spatial analyst for ArcGIS or install plugins for GrassGIS/QGIS like generalizer/simplify. Just google how to use each package for tutorials/videos on GIS users who use them.
https://plugins.qgis.org/plugins/generalizer/
https://plugins.qgis.org/plugins/simplipy/
Other options would be to use programming/scripting like in R or Python to do the same generalisation with their own specific libraries but I find the GIS packages will be far quicker to process them.
Unfortunately there is only so much you can generalise/simplify, every iteration of this will erode the quality and accuracy of your boundaries so you have to make the call on how much you want it to simplify to and just do it by trial and error. Including a shapefile in Omniscope significantly bumps up the size of the total Omniscope iok file and affects the loading time and the amount of RAM your machine will also determine what loads.]]>
Caution: attached postzone map might not be free, it's googled...]]>
Was having a play with Geometric Buffers today and these are great! The only issue is that what I really want to see is a Map Visual but also the map summarised in a table.
I have a buffer layer of 260 points across the UK with a 10km radius. This is overlayed on to another data file of 400,000 individual address with different classifications, i.e. Pubs, Shops etc. You can construct a pivot table along the Map that when you select an area the pivot table is changed to that area. However, what I want is a pivot table that is all the areas listed and what split of Pubs to shops are in their areas.
Does this make sense? I think you need to have the Buffer file as part of the dataset but when you have overlapping areas this proves difficult.
Any ideas or tips would be great.
Danny]]>
In this example we are going to analyse data relating to cities in the USA and add an overlay of USA state areas to improve the visualisation.
ATTACHMENTS
- USA cities.zip - Contains an Esri Shapefile of USA city data, including latitude/longitude locations.
- USA states.zip - Contains an Esri Shapefile of USA state areas.
INSTRUCTIONS
- Download and unzip "USA cities.zip" and "USA states.zip".
- Create a new file in Omniscope and go to DataManager.
- First we need to load the USA cities data into Omniscope. Drag "USA_cities.shp" on to the DataManager workspace.
- Expand the block and next to "Overlay identifier", select "CITY". This means that Omniscope will use the city name to uniquely identify every record.
- Next to "Metadata fields included", select all fields. This will load all meta-data from the Esri Shapefile.
- Execute the block and load it into Omniscope.
- Now that the city data is loaded we can begin exploring it. Go to the 'DataExplorer' tab.
- Change the Bar/line View to a Map View, which will display the location of each city.
- The cities are now visible on the map but we can add an overlay of the USA state areas to summarise the data. Drag the "USA_states.shp" onto the Map View and select 'Open as a map overlay'.
- The next dialog allows you to customise how the data will be imported. This contains the same options you saw in the DataManager block earlier. Next to "Overlay identifier", select "STATENAME" to use USA state names as unique identifiers for each area in the file. Next to "Metadata fields included", select all fields. This will load all meta-data from the Esri Shapefile. Click "OK".
- Now we need to change the colour of the states. In the Map View toolbar, go to the "Overlay" menu and select the downward pointing arrow next to "USA_state.shp". This will display a panel containing advanced options for your overlay layer.
- Under "Outline", change "Colour mode" to "Fixed" and then choose a new colour from the new drop-down. This will update the outline colours of all USA state areas. More information on colour modes is available at the end of these instructions.
- Under "Fill", change "Colour by (overlay dataset)" to "Subregion". This uses the "Subregion" field in your overlay dataset (USA_state.shp) to generate category colours for each area.
- You can also use data from your currently loaded dataset (USA_cities.shp) to colour the USA state areas. To do this use the same overlay configuration menu and under "Fill", change "Colour mode" to "By current dataset".
- You'll notice there is a message 'No "Linked data" field selected for the current dataset.'. For Omniscope to link these two datasets together we need a common field with values found in both datasets. Our sample SHP files both contain state names with common values so select "State" next to the "Join on (current dataset)" control to configure the link. We now have a basic relationship configured between the USA_cities dataset and the USA_state dataset.
- You might have noticed that some states, e.g. Wyoming, have disappeared from the map. By default after you have created this link between 2 datasets, Omniscope will not display any overlay areas which do not have related data in your currently loaded dataset. To disable this behaviour uncheck "Filter overlay data on join".
- The overlay is now configured to display state areas coloured by the number of linked records in the cities dataset. You can colour by a field in the current dataset by selecting from the drop-down next to "Colour by (current dataset)". Select "Pop2007" to use the population statistics for 2007. You can choose a function to summarise all the population statistics by selecting from the "Function" drop-down but we will leave this as "Sum".
- Your map is now displaying each USA state coloured by the total population across all cities within its area. Notice how California, Texas and New York all have same colour yet they have total populations of ~17.2m, ~10.5m and ~9.1m respectively. This is because we are using the original colour range from the "Pop2007" field which ranged from ~100k to ~8.3m. Any values above or below these limits will generate the same colour, which is misleading for our example. We can dynamically generate new colour limits by going back to the overlay "Fill" options and enabling '"Fit to data" colour range'.
ADVANCED
In the Map View overlay functionality you will come across terms like "Overlay ID", "Overlay identifier", "Data type". Below are explanations for each of these terms and how they are used by Omnsicope. If you are using KML or Shapefile sources then most of these fields will be configured automatically.
Data types
- Point - A dataset containing pairs of latitude/longitude coordinates, e.g. a shop location.
- Line - A dataset containing connected, and unfilled/unclosed, chains of points, e.g. a hurricane path or road.
- Area - A dataset containing connected, and filled/closed, chains of points, e.g. a post code area or country border.
Colour modes
- Fixed - Choose a single colour which will be applied to all data.
- By current dataset - Generate colours using a field from the currently loaded Omniscope dataset.
- By overlay dataset - Generate colours using a field from the embedded overlay dataset.
Fields
- Overlay identifier - Field containing unique identifiers for each point, line or area in the input dataset, e.g. a field containing country names or postal area codes.
- Overlay sub-path identifier - This is used internally within Omniscope to identify nested geometries within each shape. For example a shape for the United Kingdom could not be drawn with one continuous line. We use this field to breakdown the records into numerous islands which make up the country.
- Longitude - Field containing WGS84 longitude values.
- Latitude - Field containing WGS84 latitude values.
In this example we are going to create a spider plot linking the company's distribution centres to each of the customers awaiting a delivery.
ATTACHMENTS
- Customer deliveries.csv - A CSV file containing the location's of customers and the name of the distribution centre where the delivery will originate.
- Distribution centres.csv - A CSV file containing the location's of a company's distribution centre..
INSTRUCTIONS
- Open "Customer deliveries.csv" in Omniscope.
- Change the Bar/line View to a Map View, which will display the location of each customer.
- Now we need to add the second CSV file as an additional layer in the Map View. Drag "Distribution centres.csv" onto the Map View and select 'Open as a map overlay'.
- The next dialog allows you to customise how the data will be imported. This contains the same options you would see in a DataManager block for the same file type. Click "OK".
- Now we can configure how the Map View should interpret the distribution centre data. We need to tell Omniscope what type of data the file contains, e.g. point, line or area, and also which fields contain latitude/longitude values etc. See the ADVANCED section at the end of this post for more information. For now just configure the settings to match the screenshot shown below and click "OK".
- The distribution centres are now displayed alongside the curent dataset's customer locations but they all look the same, so we need to customise the view. In the Map View toolbar, go to the "Label" menu and select "Customer" to add some labels.
- Now we need to change the colour of the distribution centres. In the Map View toolbar, go to the "Overlay" menu and select the downward pointing arrow next to "Distribution centres.csv". This will display a panel containing advanced options for your overlay layer.
- Click on "Fixed colour" and choose another colour to update the distribution centres.
- Click on "Spider plot" to begin enabling spider plots. You'll notice there is a message 'No "Linked data" field selected for the current dataset.'. In order for Omniscope to link these two datasets together we need a common field with values found in both datasets. Our sample CSV files both contain a "Distribution Centre" field with common values so select that next to the "Join on (current dataset)" control. We now have a basic spider plot connecting each distribution centre to it's respective customers.
- You might have noticed that some distribution centres have disappeared from the map. By default after you have created this link between 2 datasets, Omniscope will not display any overlay points which do not have related data in your currently loaded dataset. To disable this behaviour uncheck "Filter overlay data on join".
ADVANCED
In the Map View overlay functionality you will come across terms like "Overlay ID", "Overlay identifier", "Data type". Below are explanations for each of these terms and how they are used by Omnsicope. If you are using KML or Shapefile sources then most of these fields will be configured automatically.
Data types
- Point - A dataset containing pairs of latitude/longitude coordinates, e.g. a shop location.
- Line - A dataset containing connected, and unfilled/unclosed, chains of points, e.g. a hurricane path or road.
- Area - A dataset containing connected, and filled/closed, chains of points, e.g. a post code area or country border.
Fields
- Overlay identifier - Field containing unique identifiers for each point, line or area in the input dataset, e.g. a field containing country names or postal area codes.
- Overlay sub-path identifier - This is used internally within Omniscope to identify nested geometries within each shape. For example a shape for the United Kingdom could not be drawn with one continuous line. We use this field to breakdown the records into numerous islands which make up the country.
- Longitude - Field containing WGS84 longitude values.
- Latitude - Field containing WGS84 latitude values.
The following example assumes that you already have an iGeolise Travel Time account. You can sign up for an account by clicking on "iGeolise account help" in the DataManager block noted below and following the link.
ATTACHMENTS
- Customer deliveries.csv - A CSV file containing the deliveries to be made from a company's distribution centre to various customers across London.
INSTRUCTIONS
- Create a new file in Omniscope and go to DataManager.
- Drag "Customer deliveries.csv" onto the DataManager workspace.
- Create a new "iGeolise Travel Time" operation and connect it to your workflow. This block will use the point data from the CSV and generate routes from the distribution centre to each customer location.
- Enter your iGeolise Travel Time application ID and key.
- Choose the "Routing" service.
- Change the "Transport type" to "Driving" so the service will calculate results based routes which the company's delivery vans can travel on.
- Enter a maximum journey duration, e.g. 60 minutes. Any destinations which cannot be reached in that time will be filtered out.
- Go to the 'Fields' tab to configure. We need to configure which fields this block should use to identify the origin and destination of each route:
- Point identifier - Field containing unique identifiers for each point in the input dataset. In our example this would be the names of the distribution centres or customers.
- Longitude - Field containing WGS84 longitude values.
- Latitude - Field containing WGS84 latitude values.
- Execute the block, load the data into Omniscope and then drag a Map View into Omniscope to see the routes.
ATTACHMENTS
- "UK major cities.csv" - A CSV file containing the locations of major cities in the United Kingdom.
- "English postal areas.kml" - A KML file containing the boundaries of all postal areas in England.
INSTRUCTIONS
- Create a new file in Omniscope and go to DataManager.
- Drag "UK major cities.csv" and "English postal areas.kml" onto the DataManager workspace.
- In the "English postal areas.kml" block, next to "Overlay identifier", select "Name". This means that Omniscope will use the postal area names to uniquely identify postal area shapes in the data.
- Create a new "Geometric filter" operation. This block requires first a point dataset and second an area dataset to function. So connect the "UK major cities.csv", then the "English postal areas.kml" (please note that the point data must be connected first).
- Select a Filter Type:
- All points (don't filter) - All points will be returned from the block. Use this in conjunction with 'Add "Areas" field' (see next step).
- Points inside any area - Keep only the points which fall inside at least one area shape.
- Points outside all areas - Keep only the points which do not fall inside any area shape.
- Check 'Add "Areas" field' to add a new field and merge the "Overlay identifier" from the Area dataset into your new Point dataset. In our example this will add the postal area codes from the "English postal areas.kml" into the new cities dataset.
- Execute the block, load the data into Omniscope and then drag a Map View into Omniscope to see the filtered point data.
ADVANCED
The "Geometric filter" block also contains additional controls for configuring which fields contain the latitude/longitude values and the fields for identifying individual features (shapes) in the input dataset. If you are using KML or Shapefile sources then these fields will be configured automatically. Below are explanations for each option:
- Overlay identifier - Field containing unique identifiers for each point or area in the input dataset, e.g. a field containing country names or postal area codes.
- Overlay sub-path indentifier - This is used internally within Omniscope to identify nested geometries within each shape. For example a shape for the United Kingdom could not be drawn with one continuous line. We use this field to breakdown the records into numerous islands which make up the country.
- Longitude - Field containing WGS84 longitude values.
- Latitude - Field containing WGS84 latitude values.
Geometric buffers can be applied to 3 types of shape data in Omniscope;
- Point - A dataset containing pairs of latitude/longitude coordinates, e.g. a shop location.
- Line - A dataset containing connected, and unfilled/unclosed, chains of points, e.g. a hurricane path or road.
- Area - A dataset containing connected, and filled/closed, chains of points, e.g. a post code area or country border.
Below are instructions for buffering these 3 types of shape data. Please download and extract the contents of the ".zip" file:
POINT
- Create a new file in Omniscope and go to DataManager.
- Drag "United Kingdom cities.csv" onto the DataManager workspace. This contains 5 major cities in the United Kingdom with latitude/longitude coordinates.
- Connect a "Geometric buffer" operation to the workflow.
- Choose "Point" as the "Input data type". (For KML and Shapefile sources, this will be configured for you.)
- Enter a distance (km) to buffer around each input point. By reducing the "Quality" you will increase the speed of this block but the output dataset will contain less records and the buffered outline will not be as smooth.
- Check 'Include metadata' if you would like the output dataset to contain all other fields found in the input, i.e. "Country name".
- Execute the block, load the data into Omniscope and then drag a Map View into Omniscope to see the buffered points.
LINE
- Create a new file in Omniscope and go to DataManager.
- Drag "Hurricane Katrina.csv" onto the DataManager workspace. This contains coordinates for the path of Hurricane Katrina in 2005.
- Connect a "Geometric buffer" operation to the workflow.
- Choose "Line" as the "Input data type". (For KML and Shapefile sources, this will be configured for you.)
- Enter a distance (km) to buffer around the line (hurricane path). By reducing the "Quality" you will increase the speed of this block but the output dataset will contain less records and the buffered outline will not be as smooth.
- Execute the block, load the data into Omniscope and then drag a Map View into Omniscope to see the buffered line.
AREA
- Create a new file in Omniscope and go to DataManager.
- Drag "United Kingdom.kml" onto the DataManager workspace. This contains coordinates defining a simplified boundary of the United Kingdom..
- Connect a "Geometric buffer" operation to the workflow.
- Ensure "Area" is selected as the "Input data type". (For KML and Shapefile sources, this will be configured for you.)
- Choose your method of buffering:
- Expand area - The original area plus an outside perimeter.
- Contract area - The original area less an inside perimeter.
- Outside perimeter - The area within a certain distance outside of the perimeter.
- Inside perimeter - The area within a certain distance inside of the perimeter.
- Outside + inside perimeter - The area within a certain distance either side of the perimeter.
- Enter a distance (km) to buffer around the United Kingdom. By reducing the "Quality" you will increase the speed of this block but the output dataset will contain less records and the buffered outline will not be as smooth.
- Execute the block, load the data into Omniscope and then drag a Map View into Omniscope to see the buffered line.
ADVANCED
The "Geometric buffer" block also contains additional controls for configuring which fields contain the latitude/longitude values and the fields for identifying individual features (shapes) in the input dataset. If you are using KML or Shapefile sources then these fields will be configured automatically. Below are explanations for each option:
- Overlay identifier - Field containing unique identifiers for each point, line or area in the input dataset, e.g. a field containing country names or postal area codes.
- Overlay sub-path identifier - This is used internally within Omniscope to identify nested geometries within each shape. For example a shape for the United Kingdom could not be drawn with one continuous line. We use this field to breakdown the records into numerous islands which make up the country.
- Longitude - Field containing WGS84 longitude values.
- Latitude - Field containing WGS84 latitude values.
ATTACHED FILES
- "postcode areas.shp" (inside ZIP) Contains geometries for each feature (shape) in the Esri Shapefile
- "postcode areas.dbf" (inside ZIP) Contains meta-data for each feature (shape) in the Esri Shapefile
- "postcode areas.prj" (inside ZIP) Contains the coordinates' projection, e.g. UTM, WGS84.
INSTRUCTIONS
- Download the attached ".zip" file and extract the ".shp", ".dbf" and ".prj" files into the same folder.
- Create a new file in Omniscope and go to DataManager.
- Drag the "postcode areas.shp" file onto the DataManager workspace (or drag a "Data file" block onto the workspace and browse to the "postcode areas.shp" file). The ".dbf" and ".prj" files will be automatically detected.
- The Esri Shapefile DataManager block contains 2 configurable options
- "Overlay identifier" - The field which contains unique identifiers for each feature (shape) in the file. In our Shapefile this would be the names of the postal areas, e.g. AB for Aberdeen, SW6 for South West London. If your data does not include such a field then selecting "Create new field" will allow Omniscope to generate internal unique identifiers using a field named "Overlay ID".
- "Meta data fields included" - The names of all fields which will be loaded from the Esri Shapefile into the block.
- Load the data into Omniscope then drag a Map View into Omniscope to see the postcode areas.
ADVANCED
You will notice a field has also been created named "Overlay sub-path ID". This is used internally within Omniscope to identify nested geometries within each shape. For example a shape for the United Kingdom could not be drawn with one continuous line. We use this field to breakdown the records into numerous islands which make up the country.]]>
ATTACHED FILE
- "postcode areas.kml" (inside ZIP) Contains geometries and meta-data for each feature (shape)
INSTRUCTIONS
- Download the attached ".zip" file and extract the ".kml" file.
- Create a new file in Omniscope and go to DataManager.
- Drag the "postcode areas.kml" file onto the DataManager workspace (or drag a "Data file" block onto the workspace and browse to the "postcode areas.kml" file).
- The KML DataManager block contains 3 configurable options:
- "Overlay identifier" - The field which contains unique identifiers for each feature (shape) in the file. In our KML file this would be the names of the postal areas, e.g. AB for Aberdeen, SW6 for South West London. If your data does not include such a field then selecting "Create new field" will allow Omniscope to generate internal unique identifiers using a field named "Overlay ID".
- "Meta data fields included" - The names of all fields which will be loaded from the KML file into the block.
- "Shape types included" - KML files can contain a combination of points, lines and areas. Use this option to filter which shape types you want to load.
- Load the data into Omniscope then drag a Map View into Omniscope to see the postcode areas.
ADVANCED
You will notice a field has also been created named "Overlay sub-path ID". This is used internally within Omniscope to identify nested geometries within each shape. For example a shape for the United Kingdom could not be drawn with one continuous line. We use this field to breakdown the records into numerous islands which make up the country.]]>
I have been asked to produce a Map view report by our marketing team. Basically we wish to show on the map all our locations. We will also show on the same map the locations of a partner of ours. I was able to get this to show by the usual methods. But what the marketing team would like is to show a small graphic instead of the default marker. So for all our locations we show a small graphic and for all the partner we show their graphic (small logos) This was we can show the complementary locations of both our sites.
Can you let me know how I can do this?]]>
Is this supported?
Dave
]]>
You can download them from github at:
https://github.com/ucfnlam/OmniscopeDemos?source=c
Or from my blog which has webstart links when you click on the icons:
http://spatiametrics.wordpress.com/omniscope-data-visualisations/
]]>
Use this to transform geographic data imported typically from KML files or ESRI Shapefiles.
It includes the ability to convert point data such as building locations into circular areas representing a specified distance from that location (stored as a series of point records with a shared ID). You might use this to find the areas representing 5km from tube/subway stations, for example.
It also allows you to convert area (polygon) or line (polyline) data such as coastlines or zip code boundaries by enlarging outwards, subtracting inwards, or creating a "channel" either/both sides of the outline, according to a specified distance. You might use this to find the area representing 20 miles from the coast, for example.
The resulting geographic "areas" can be used as an overlay in the Map view.
Please be aware that buffering can multiply the volume of data by several 100 times, since each point can introduce many points to describe the area around the point/line. Also note that while point buffering is true to the Earth's shape, line/area buffering is done in Euclidean space, so is only accurate at the local level and becomes completely inaccurate when you have areas/lines covering a large proportion of the world. Only supports WGS84 coordinates.
]]>