Geocoding in 2.6

Mees · August 10, 2010 2:58PM

Hi there,

With the geofunctions in 2.6 we are trying to get geocodes for our database.
As we presume the number of queries is limited to 1000, we have to execute the operations multiple times.
No problem, but we are trying to limit the manual work as much as possible.

We noticed that Data Manager drops existing data output (Omniscope table, csv file etc) and recreates a new one when refreshing. In other words, it does not append records to already existing records (existing records for which we have already retrieved the geocodes), unless we use the Append operation. However, using the Append operation means we have to duplicate manually the Filter operation (that limits the number of records for us). As we need a lot of geocodes, we need to duplicate a lot and therefore must find a more efficient way for doing this.
Can you help? Can we configure DM so that it limits the number of records and adds new records to existing Omniscope output?

As an explample I have attached an iok file that represents our process with example data.

Many thanks

Mees · August 10, 2010 3:09PM

Would it be possible to add functionality to the Filter operation that enables us to limit the number of records, not based on a field but just a random number of records?
The current filter functionality requires a fieldvalue to filter on. My suggestion here is, to add a filter option for a limited number of records, e.g. 1000, randomly selected by DM.

steve · August 11, 2010 9:52AM

We are considering the possibility of adding a "random subset" operation, where you specify the number of records to keep. What would you use this for?

Mees · August 23, 2010 8:52AM

Hi Steve,

That would indeed be useful. After downloading the latest 2.6 build yesterday, I saw the Random Sample operation. Is that new? I haven't got the time yet to use it, but does it relate to the above?

Another related question is the output to e.g. csv files. Changing settings upstream in DM (eg filter settings), changes datasets and thus also output content. However, the name of the output file doesn't change but the previous file is overwritten by the new one. Would it be possible to have a flexible file naming method? Flexible meaning that it is defined by eg filter settings upstream.

Thanks, A

steve · August 24, 2010 9:46AM

Yes, "Random sample" is what you're looking for. You might also find "Input switch" useful.

Dynamically generated output filenames based upon filter settings upstream is an interesting idea but not one we'll be exploring in 2.6. You may be able to achieve the same effect using Batch Output, where you provide an IOK file containing "commands" (filter settings, output actions etc.). You will need an Enterprise license to try this.

Welcome!

Categories

Ideas Parade

Tagged