Hello Everyone,

I'm a first time poster but I'm having a significant issue with the VisualCron service that is impacting our operations. I hope to get it resolved through some mindshare. I apologize in advance for the wall of text but I wanted to be thorough up front to minimize the back and forth.

SUMMARY OF PROBLEM:

The Visual Cron service crashes randomly on a regular basis, usually lasting a few days before crashing, after a transition from a Windows managed file system to a raw CIFS share on a VNX SAN. Samba polling has been enabled which has caused a significant spike in memory usage over what it was using previously when managed by Windows.

BACKGROUND:

We originally had a virtual server that was 3.5 Terabytes large, the file server LUN part of this virtual server being hosted on a VNX 5300, which hosted all the files that were being monitored by the VisualCron service. As part of implementing a new VNX 5400, we created a CIFS share and eliminated the virtual server (running Windows) by replicating all the data from the virtual server to the new CIFS share - essentially replicating the data from the VNX 5300 to the new VNX 5400. The key difference here is that the file system was being managed by the Windows OS on the virtual server setup and the CIFS is being managed by the VNX 5400. We simply mapped the IP of the former virtual server to the CIFS share, made sure all the local usernames and passwords were mapped correctly, and that the UNC path \\servername\foldername, worked successfully.

I'd like to say we nailed it the first time but it took some testing, specifically around the local accounts, to get the data flowing correctly from the FTP server to the share that is monitored and files triggered via VisualCron. However, there were a number of crashes as we did not do a good job of documenting the local user access rights of the former server, so we had to spend time in getting that situated. Throughout the troubleshooting of these initial crashes, it was mentioned by VC support that we needed to enable Samba polling. Not sure why since Samba is not in play here. It has now been running for about 2 weeks, but since Samba polling has been configured the VisualCron system is now using up quite a bit of memory and the VC service crashes every few days with a random error stating to contact support, the only real complaint regarding an illegal character in the path.

We are contemplating putting a Windows server back in front of the CIFS share since that would take us back to a "last known good state" but I wanted to also poll this group to see if anyone had any insight. Again, I apologize for the wall of text but I wanted to be thorough so I could get the best support. I'm sure there will be questions so please give this some thought and fire away. I look forward to learning from some of you regarding this issue.

One thing to add here that I forgot to mention is that after the move to directly accessing the CIFS share the event triggers were not being triggered as data was written to the new CIFS share. So there was bad usernames and passwords at first that needed to be corrected, and then the issue with the event triggers. During that troubleshooting Samba polling was turned on. This still makes little sense to me, because this is not a SAMBA file system. Maybe this is where VisualCron goes over my head, but the SAN engineers I have talked to have stated this is a pretty common configuration for most task scheduler software out there.

One other point worth mentioning is that VisualCron is handling hundreds and thousands of trigger events during that 3 or 4 day issue with no problem before the service crashes. Thanks!

Hi,

1. which version are you running?
2. we are interested in the log files log_serverDATE.txt from the day it crashed (please send to support@visualcron.com)
3. if it is a crash on a lower level you should see an error in the event log at crash time. Please paste the information here.

---

Hi Henrik,

Thanks for the quick response. Here are my responses:

========================================
Response to #1

Version 7.1.6
Build 18671
Build date: 2014-04-16
========================================
Response to #2

The log file you are asking for is 256mb large for the 24 hour period starting 5/19 at 12:00am. I can't get this to you over e-mail, what are other options?
========================================
Response to #3 - here is the event id and detail:

Event 0, VisualCron

An unhandled error has occured (please send the information below to support@visualcron.com):

Server version: 7.1.6.18676 - (2014-04-16 04:22:35)
Protocol: 7.0.0
OS: WindowsServer2008R2DataCenter
Architecture: x64

Thread: (761)

System.ArgumentException: Illegal characters in path.
at System.IO.Path.GetFileName(String path)
at System.IO.FileSystemWatcher.MatchPattern(String relativePath)
at System.IO.FileSystemWatcher.NotifyFileSystemEventArgs(Int32 action, String name)
at System.IO.FileSystemWatcher.CompletionStatusChanged(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* overlappedPointer)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Have you checked "Use polling (SAMBA)" in the File Trigger settings?

---

Hold on, I see that we already have a support case open about this. We have an ongoing email discussion about this. I suggest installing 7.1.7 which includes the added debugging for this case.

---

Henrik,

Yes, we have an open issue but the folks that were working the issue need to focus on other areas of operations. I am now taking over the case because I believe root cause is important here.

I have e-mailed you an excerpt of the log file in question. I believe there should be enough in that log file to determine root cause. Please take a look at it and let me know what you think. Note the time stamp of the crash at 09:00:42pm. It looks like jobs kick off around 9:00pm and right at 09:00:40pm, right before the crash, there are quite a few errors that get generated.

Thanks.

Thanks, please install 7.1.7 and send us the file again if it happens again.

---

Henrik,

Please take a look at the log file I sent over to you and let me know if you see anything that we need to address that may have contributed to the crash. There are a lot of null value messages along with some tracking id error messages that are happening around that specific time frame.

Thanks.

As mentioned in the email conversation we did changes in 7.1.7 to add more logging why this happens. So, the only way to get closer to finding the problem is to install the new version.

So, hopefully, with a new log from 7.1.7 we can find the reason.

---

I know this does not solve the problem, however...

I've run into occasional, unexplained server crashes, so I've taken to adjusting the VC service settings to automatically restart.

---