Idea: Automate download (FTP) and import of zipped files?
-
Is Omniscope able to import via FTP remote data files that are zipped? I've got a CSV file that I'm trying to import using the URL module, but it doesn't seem to work.
Is importing an FTP file directly to Omniscope much quicker than downloading the file to your own PC, unzipping and then importing it as a flat file into Omniscope? File size is 500MB. This is the first time I've dealt with a file of this size within Omniscope and when I tried importing the 500MB file on my local drives, it started hanging after importing 20% or so.
I do currently run Omniscope off my laptop, so that may not have the required processing power/RAM for the larger files. -
11 Comments
-
Shaji - Currently, automatically unzipping .ZIP data files while importing the data is not possible in Omniscope. I have re-classified this as an Idea, since would not be difficult to add this feature, and it would enable totally automated periodic 'time-slicing' of zipped remote data (and archiving locally as compact IOK Timeslice files for batch appends) using the Scheduler.
If you are going to be manually downloading, un-zipping and importing very large data files, make sure you are using a machine with a 64-bit (not 32-bit) operating system (Windows). Machines with 32-bit operating systems can only address about 1.1 GB of RAM, thereby limiting the file size that can be managed in-memory on the machine. If you have already changed to 64-bit Windows, then make sure you fully-populate the RAM on your machine...RAM is now very cheap. -
Hiya - Do we have an update on this? Or even a workaround? Thanks,
-
Sherrick - Why not tell your data vendor to use Dropbox/GDrive, etc.?
Seriously, this will not be difficult....but it needs a few more agency votes...are you the only one who wants this..? If not, cut-paste the page link and circulate to others to get more votes.
One question: What should be the behavior if there are more than one file inside the zip folder...of same or different files types....is this an issue? -
Hi,
We've had a discussion as to how we see this working.
We could add the ability to select a "ZIP" file from the "Open file" menu. This would assume that the ZIP file contained only one file inside it. You would have to specify the type of file being imported (in your example above this would be "CSV"). If the ZIP file contained MORE than one file, Omniscope would show an error message.
If you wanted to open a file from a ZIP file that contained more than one files then we could add support for ZIP files in the "Batch append" operation. ZIP files are essentially a folder of files, so using the "Batch append" operation you could specify the name(s) of the files you wanted to import and the parsing options. Using the batch append operation you could choose to append several files on one go from a ZIP file or a single file.
I think both of the above would cover most use-cases when working with ZIP files. Please let us know what you think.
Welcome!
It looks like you're new here. If you want to get involved, click one of these buttons!
Categories
- All Discussions2,595
- General680
- Blog126
- Support1,177
- Ideas527
- Demos11
- Power tips72
- 3.0 preview2
Tagged
To send files privately to Visokio email support@visokio.com the files together with a brief description of the problem.