Merging data blocks when one is empty default take non empty data block

benjamin · April 21, 2011 4:04AM

Hi

I have two data source in data manager .One data source is a file and the other data source is Yahoo New Search.

Now the file contains number of records hits on a companies internet site per day.

The yahoo news search block retrieves yahoo news results for the companys website.I then aggregate the output of the yahoo news block by publich date and do a count number of records on this aggregation .

Then i merge the two outputs of the block sources together by date.

Idea being i will have number of visit to a company website a day versus the number of news results for that company in that day.

The only problem is , that the yahoo news search returns "no data" if it cannot find a news search with the companies name input, thus all subsequent blocks fail to populate .As a result i get a error message on the merge block and no data is loaded into omniscope.

Is there any way to oversoem this problem.

I would suggest some sort of override box in data blocks which takes first input block if one return null/no data.

Thanks

chris · April 21, 2011 4:23AM

At the moment all merging feed operations (Yahoo news/web search etc) will generate an error if no results are found. As you mentioned this will cause any subsequent downstream blocks to error. An easier solution than the one you suggested would be to simply return a zero-row dataset if no results were found. This would mean the columns would be intact, but there was no data. Joins, appends and other downstream operations would be unaffected. What do people think of this idea? Should this be an option or the default behaviour?

benjamin · April 21, 2011 4:35AM

I think this should be default behaviour (Or at least an option) as apriori you will not know if any given word will return any search results in yahoo news.Especially if you are using parameterised query from another omniscope block as input into this block.

steve · April 21, 2011 5:23AM

On the one hand, you might say that "no results" should equate to "zero records" without an error. This would be consistent with the Join/Merge block.

On the other hand, if you have hard-coded the input value to a web service, and something changes with the web service, you want the problem flagged.

So an option in merging feeds would be useful here, but I'm not sure what the default should be. Probably to allow zero results. Are there any other blocks which need this option?

chris · April 21, 2011 6:27AM

We should probably provide this option in any online source that returns a pre-determined set of fields. So initially this would be things like Email, Website map, Twitter and Yahoo searches.

steve · April 21, 2011 7:30AM

In fact all Merging Feeds, and the option should allow no results by default.

Welcome!

Categories

Ideas Parade