We are getting frequent System.OutOfMemoryException errors when someone attempts to connect to the VC server.
The first instance of the problem on this server occurred after a job encountered the following error:
11/18/2008 8:03:09 PM Debug Task (425) was removed from processlist: Delete file from FTP server
11/18/2008 8:03:09 PM Err An unhandled error has occured (please send the information below to
support@visualcron.com):
Server version: 4.9.26.36962 - (2008-09-07)
Protocol: 5.0.0
OS: Windows2003
System.NullReferenceException: Object reference not set to an instance of an object.
at B60fA89zU8IFKUapjfk.1BFwPr9Bf7vKsr94yyi.Z9e9bTS2h()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
11/18/2008 8:03:09 PM Debug Starting restarter...
The delete task did delete the file successfully, and it was the final task in the job; I believe the error was likely when the job was attempting to read the next execution time. This error caused a VC restart.
Then, when someone attempted to connect immediately after the restart, the following error was received for the first time:
11/18/2008 8:07:33 PM Info User "UserName" - Logged in succesfully through Socket (c025739a-18dc-4580-80a8-2a3f9ab920b3)
11/18/2008 8:07:33 PM Debug Updating login status for: c025739a-18dc-4580-80a8-2a3f9ab920b3
11/18/2008 8:07:33 PM Comm IN: SERVERSETTINGSREQ
11/18/2008 8:07:33 PM Comm IN: LICENSEREQ
11/18/2008 8:07:35 PM Err Select case ex: System.InvalidOperationException: There was an error generating the XML document. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Xml.XmlUtf8RawTextWriter.FlushBuffer()
at System.Xml.XmlUtf8RawTextWriter.WriteRawWithCharChecking(Char* pSrcBegin, Char* pSrcEnd)
at System.Xml.XmlUtf8RawTextWriter.WriteRaw(Char[] buffer, Int32 index, Int32 count)
at System.Xml.XmlRawWriterBase64Encoder.WriteChars(Char[] chars, Int32 index, Int32 count)
at System.Xml.Base64Encoder.Encode(Byte[] buffer, Int32 index, Int32 count)
at System.Xml.XmlRawWriter.WriteBase64(Byte[] buffer, Int32 index, Int32 count)
at System.Xml.XmlWellFormedWriter.WriteBase64(Byte[] buffer, Int32 index, Int32 count)
at System.Xml.Serialization.XmlCustomFormatter.WriteArrayBase64(XmlWriter writer, Byte[] inData, Int32 start, Int32 count)
at System.Xml.Serialization.XmlSerializationWriter.WriteElementStringRaw(String localName, String ns, Byte[] value, XmlQualifiedName xsiType)
at System.Xml.Serialization.XmlSerializationWriter.WriteElementStringRaw(String localName, String ns, Byte[] value)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterMessageClass.Write101_TaskStatsClass(String n, String ns, TaskStatsClass o, Boolean isNullable, Boolean needType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterMessageClass.Write242_TaskClass(String n, String ns, TaskClass o, Boolean isNullable, Boolean needType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterMessageClass.Write245_JobClass(String n, String ns, JobClass o, Boolean isNullable, Boolean needType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterMessageClass.Write320_MessageClass(String n, String ns, MessageClass o, Boolean isNullable, Boolean needType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterMessageClass.Write321_MessageClass(Object o)
--- End of inner exception stack trace ---
at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id)
at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces)
at eNlCfYiiah6U6FAayIS.m1445miysM1ppEhyhsX.NUSeZCicXSfGxO5wKr2.XRulHvi41qn71GY9ibC.MKgR2987K(Object )
at fmyPQDsh17VlFO4rKh.DD4SN7GmSiwWpIhwSK.AW1JrSuZG(MessageClass , String )
at bJXxxM9hDJ9krEhTMnu.7uqhsl9vbtsMjeTJcdg.p7q32ACU1(String )
at KRgyIKlDBYvCSdZocCw.4xWNW6lYlCw28RbNAML.88OWMmmUG(String , MessageClass )
at KRgyIKlDBYvCSdZocCw.4xWNW6lYlCw28RbNAML.RVfRb3gOL(String , MessageClass , Byte[] )
11/18/2008 8:08:49 PM Debug Cleanup time?
Both the error associated with the job; and the systemoutofmemory error associated with a login have become very persistent.
For the error associated with the job, I attempted to recreate it from scratch twice (no cloning.) But it still gets the error message and causes VC to restart. I also tried eliminating the last delete task - the problem occurs no matter what the final task is. There is nothing unusual about the job - it is performing the same activities that we have many other jobs doing as well.
For the outofmemory error message at login, this seems to be worsening with time - it can occur anytime someone logs in, not just when VC has had a forced restart; and people have reported being unable to connect to VC at all (locally or remotely) for an extended period of time. Also, it does not seem to be helped by restarting the VC windows service.
I am wondering if it is possible that a settings file has become corrupt? We keep daily backups; however, we would lose a distressing amount of work to restore the settings, especially for the jobs file, from prior to the beginning of the problem on 11/18. My hope is that if you think a settings file needs to be replaced, it might be possible to identify which one is the most likely culprit (hoping that it is something other than the jobs file.) Or if you have any other minimally disruptive ideas about things to try.
One other comment is that the current size of the jobs file is 125M - is there an upper limit to the size of the xml file that VC can handle efficiently?