We have a set of extracts that generate very large files that take forever to write to the network. I'm looking for tips on doing it differently.
The jobs have 3 tasks. Read SQL code from file/execute Oracle SQL/write output to file.
Some files contain a very large number of records (2.5 million) and these take a couple of hours to finish. I can run the same queries in Toad and they run in no time, thus I suspect the issue is somewhere in the SQL caching the output or the write task reading that and writing. It's been running over an hour right now and still hasn't started writing to the network drive.
Has anyone else overcome a similar issue?

May be related: If have have several of these running at the same time and kill the jobs, we find that the VC program will crash and we get disconnected if we restart the jobs and let them run for a while. VC needs to be manually killed and restarted from the server. Is it possible we have a configuration set wrong and are using memory incorrectly or are not clearing memory on a crash? If you can point me to the right place to look or troubleshoot I'd be grateful.

Gary

VC 8.2.8 on Windows 7 64 bit clients and Windows server 2012 VMs with 64GB memory

Update: Naturally right after I posted this a coworker showed me the "clear memory" button on the server info screen, which we will now do after killing jobs/tasks if we have to. Hopefully that is best practice for that scenario.

It would be interesting if you tested 8.3.0 as we have improved SQL Task execution time. How long does that Task run right now?

https://www.visualcron.c....aspx?g=posts&t=7604 

---

It runs ~2.5 hours in VisualCron, including the time to write to the file. I watched the output folder, and it ran for at least 45 minutes before I stopped watching it and it had not yet started writing to the file. In toad, it runs for just a few minutes.

I'll see if we can stand up a test instance for 8.3.0, we just purchased additional licenses.

It runs ~2.5 hours in VisualCron, including the time to write to the file. I watched the output folder, and it ran for at least 45 minutes before I stopped watching it and it had not yet started writing to the file. In toad, it runs for just a few minutes.

I'll see if we can stand up a test instance for 8.3.0, we just purchased additional licenses.

Thanks, please let us know!

---

Ok, set up the test server today. Visualcron 8.3.0 is the only thing on it. Still took 1.5 hours. The same query in Toad took 10 minutes to run and write to the same folder on the network (2.5 million rows to a flat file). If you would like to see any settings or anything, we can arrange a remote session, just let me know. I sure would be willing to work with you folks.
Oh, when the write was finished I noticed via the task manager that 29% of memory was still in use. VisualCron server was using 3.1GB of memory. I pressed the clear memory button on the server info tab and memory usage went to 10%, VisualCron server using 10 MB. Maybe a clue?

Thanks for your attention,
Gary

We are limited to what we can do because the way output is constructed.

• We can improve the query. This has already been done.
  
• When the query is complete we store the output to memory. There are some time related to modifying the output as you may want specific output format (see output column).
  
• Next step is the File write Task which just writes the whole memory chunk to file.

The question is what takes time for you. Take a screenshot of the Job log so we can see each Task.

---

The hour and a half is all in the SQL task. Writing to the file is only taking a minute and 14 seconds. Let me turn on extra task debugging and see what we get.

log file sent to support e-mail. Thank you.