Three times in the past week, we've had an installation of VC lose its jobs. It's been a different installation affected each time. The first two times, only the jobs disappeared. The last one, however, lost jobs, conditions and connections, although user variables stayed.

Looking in the VisualCron\Settings folder, I see these files listed:

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 8/25/2017 3:07 AM 1310922 jobs.0jdxntxb.mfu.xml
-a---- 9/25/2017 12:51 AM 0 jobs.lnev4uz3.z4v.xml
-a---- 9/25/2017 3:05 AM 118 jobs.xml
-a---- 9/25/2017 3:05 AM 118 JobsFailed20170925.xml

Jobs.xml should be about 1.2 MB in size.

It would appear that a low-virtual-memory condition had something to do with this, because this is in the server log just before midnight on the 24th:

9/24/2017 11:58:55 PM Info Job started: AccessToXml
9/24/2017 11:58:55 PM Info Task started(Write file): Create log file, if necessary (403294)
9/24/2017 11:58:55 PM Info Task completed (Success)->'Create log file, if necessary' (403294)
9/24/2017 11:58:55 PM Info Task started(Create folder): Create base folder, if necessary (403295)
9/24/2017 11:58:55 PM Info Task completed (Success)->'Create base folder, if necessary' (403295)
9/24/2017 11:58:55 PM Info Task started(Create folder): Create output folder, if necessary (403296)
9/24/2017 11:58:55 PM Info Task completed (Success)->'Create output folder, if necessary' (403296)
9/24/2017 11:58:55 PM Info Task started(Write file): Write PowerShell parameters to log file (403297)
9/24/2017 11:58:55 PM Info Task completed (Success)->'Write PowerShell parameters to log file' (403297)
9/24/2017 11:58:55 PM Info Task started(PowerShell): Get Data from FSSMain (403298)
9/24/2017 11:58:55 PM Err Error at starting process (C:\Program Files (x86)\VisualCron\\PowerShell\TaskPowerShell.exe): System.ComponentModel.Win32Exception (0x80004005): The paging file is too small for this operation to complete
at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.MHPBMJGCOCLBHDHAAMMLFOPNPEIFDLGGMLMM.DGKFNFEIJEBKOPHPLDDNBGAOOIKBNNOGOGFH(NOEMICIHJDCELPNILBDNIEPEHLIIGGIANCCC ) in C:\sourcefiles\code\VisualCronService\Jobs\TaskProcesses\Process\ExecuteHelper\clsExecuteHelper.vb:line 845

It appears that the service restarted soon after that, because the first entries in the server log for the 25th appear to be all startup information. Then there's this:

9/25/2017 12:00:25 AM Debug Loading jobs
9/25/2017 12:00:27 AM Err Error loading jobfile, new file was copied to: C:\Program Files (x86)\VisualCron\settings\JobsFailed20170925.xml, error: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderJobClassArray.InitIDs()
at System.Xml.Serialization.TempAssembly.InvokeReader(XmlMapping mapping, XmlReader xmlReader, XmlDeserializationEvents events, String encodingStyle)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.GHAMFGCEDFOEIDCMEENNAFIOCECPBDKLJMLG.AFPAJLJGMPPLGGPABDOPLIPDADFDPIMBMKKE(String , Boolean , Boolean , Boolean ) in C:\sourcefiles\code\VisualCronService\Jobs\apiJob.vb:line 488, inner exception:Exception of type 'System.OutOfMemoryException' was thrown.

I can get the low virtual memory situation creating a problem for the service, but why should the jobs be lost because of that?

Looking at the times it seems like you have some updates or auto-reboot going on at that time. Please compare with the event log what happens at that time.

---

Looking at the times it seems like you have some updates or auto-reboot going on at that time. Please compare with the event log what happens at that time.

I don't think I see anything like that--just low memory events and the VC service restarting. Here's the system log, in case you can make anything of it.
  BP13618_System.zip (69kb) downloaded 26 time(s).

1. Which version of VC are we talking about?
2. Do you have any dmp file in the log folder in the installation folder?

---

[Dump removed]

Here is the dmp file. Thanks.

Thanks, from what we can see the VisualCron service experience a Out of memory error right at startup. Is that what you experience too? Can your reproduce this, how is the memory usage in general on the server?

---

In general, about 1.2 GB of the 4.0 GB on the server are available most times. I've rebooted the machine, and so far there have been no low virtual memory events and no restarts of the VisualCron service. I guess unless it happens again there is not much else I can look at.

It happened again last night. A whole lot of low-virtual-memory errors in the Windows System Log, and out-of-memory exceptions in the Windows Application log. The VC service restarted many times. I've collected the VC server logs, dmp files and Windows System and Application logs, in case you think there's any point in looking at them.

I think we've got a couple of installations that are under-spec'ed, now that I look at them. They could use more RAM, definitely. What is most worrisome, though is the loss of configuration during these events. This last time, the VC server lost its jobs, conditions and connections, but kept its variables. I restored them from a master config backup, so it's working again, but I don't want to keep telling my colleagues this will probably keep happening.

It happened again last night. A whole lot of low-virtual-memory errors in the Windows System Log, and out-of-memory exceptions in the Windows Application log. The VC service restarted many times. I've collected the VC server logs, dmp files and Windows System and Application logs, in case you think there's any point in looking at them.

I think we've got a couple of installations that are under-spec'ed, now that I look at them. They could use more RAM, definitely. What is most worrisome, though is the loss of configuration during these events. This last time, the VC server lost its jobs, conditions and connections, but kept its variables. I restored them from a master config backup, so it's working again, but I don't want to keep telling my colleagues this will probably keep happening.

You can potentially lose settings if settings are saved during a crash - but it requires very special circumstances. I think it is important that you address the virtual server issues.

---

You can potentially lose settings if settings are saved during a crash - but it requires very special circumstances. I think it is important that you address the virtual server issues.

Whatever these special circumstances are, they are happening often within our installations. I just had to restore the jobs on a PC that had suffered a power outage. This time, the conditions, connections and variables were all still intact. I had to do three others over the weekend.