Hi

Our VCservice has crashed out with an unhandled exception, version 6.1.7.
Is this a known issue? If I upgrade to 6.1.8 will it fix the problem?
To me it looks like the automatic settings backup was happening. It may help you to know I am using DFS on the location of the settings file.


Thanks in advance. Exception text follows ...


+ System

- Provider

[ Name] .NET Runtime

- EventID 1026

[ Qualifiers] 0

Level 2

Task 0

Keywords 0x80000000000000

- TimeCreated

[ SystemTime] 2013-01-24T23:00:01.000000000Z

EventRecordID 31824

Channel Application

Computer xxxxxxxx

Security


- EventData

Application: VisualCronService.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.AccessViolationException Stack: at SevenZip.IOutArchive.UpdateItems(SevenZip.ISequentialOutStream, UInt32, SevenZip.IArchiveUpdateCallback) at SevenZip.SevenZipCompressor.CompressFilesEncrypted(System.IO.Stream, Int32, System.String, System.String[]) at SevenZip.SevenZipCompressor.CompressFilesEncrypted(System.String, Int32, System.String, System.String[]) at SevenZip.SevenZipCompressor.CompressFilesEncrypted(System.String, System.String, System.String[]) at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.DNIKJOPGFECCBCCDJLGCJELOJOLMFDAAMPOL.CJDKLHHMNNKKMJLPIGLILLMCPGNFMBFGGFKK(SettingsT, System.String, SevenZip.SevenZipCompressor, VisualCron.ExportImportProgressClass, System.String) at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.DNIKJOPGFECCBCCDJLGCJELOJOLMFDAAMPOL.OAMOMCLJLAKHHOJNBHFGLIFLBOLAILKLIPCN(Boolean ByRef, Boolean, System.Collections.Generic.List`1<SettingsT>, System.String, Boolean, VisualCron.ExportImportProgressClass ByRef) at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.DOJBHEHOLFKIKHOAGLFFCAKBACKGFDABPMMC.OAMOMCLJLAKHHOJNBHFGLIFLBOLAILKLIPCN() at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.DOJBHEHOLFKIKHOAGLFFCAKBACKGFDABPMMC.Run() at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart()

I see that this happened when the Backup Tasks job were run. Can you reproduce this?

---

Hi Henrik

I will be able to test this later after my delayed daily jobs complete.

What I can tell you is that this has happened twice this week, Monday and Thursday at 11pm, and the first time it happened on all 3 of our production servers, this time just one of the servers was affected. It's serious for our business because none of our daily jobs ran at the correct time.

As a workaround I have disabled the backups.

John

Have you made any changes to that backup Task? Like where it is stored? Added a Credential?

---

OK, I retested running manually about 20 times no problem.
I tested with a trigger several times, no problem.
The error has only occurred on the normal 11pm trigger.

I have changed the path and filename used by the backup task.
The file is on a DFS disk, i.e. it replicates between 3 geographical locations using microsoft Distributed File System. The sites are connected by VPN.
The disk is also external to the server that uses it, either a SAN or iSCSI, I'm unsure precisely.

John

Ok, so, it seems like something happens with the disks during that time.

---

Ok, so, it seems like something happens with the disks during that time.

It's possible. Having the vc service stop and fail to restart is undesirable. I'd expect it to handle the exception.

Yes, it depends on what level the error happens. If this happens on a lower level than VisualCron operates. We will check this.

---

I have just encountered a similar error while backing up VisualCron tasks. The task backs up at midnight every night. VisualCron has been running unattended without error until this past Wednesday and Sunday. The task actually appears to complete (the backup file is created) and then crashes (8 seconds after midnight) afterwards. Thursday, Friday, and Saturday job/tasks, in between, did not cause the same result and ran without incident.

Checked with network admin - No system backup runs at this time, no windows updates during this period of time.


Windows 2008 R2
Visual Cron 6.1.8.16372
Virtual Machine (VMWare)

************************
From Windows Event Viewer

[ Name] .NET Runtime

- EventID 1026
[ Qualifiers] 0
Level 2
Task 0
Keywords 0x80000000000000
- TimeCreated
[ SystemTime] 2013-06-02T04:00:08.000000000Z
EventRecordID 2293
Channel Application
Computer vc.local [i have obfuscated]

Security

- EventData

Application: VisualCronService.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.AccessViolationException Stack: at SevenZip.IOutArchive.UpdateItems(SevenZip.ISequentialOutStream, UInt32, SevenZip.IArchiveUpdateCallback) at SevenZip.SevenZipCompressor.CompressFilesEncrypted(System.IO.Stream, Int32, System.String, System.String[]) at SevenZip.SevenZipCompressor.CompressFilesEncrypted(System.String, Int32, System.String, System.String[]) at SevenZip.SevenZipCompressor.CompressFilesEncrypted(System.String, System.String, System.String[]) at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.DEIOCOLCLLOAPDPOCJGIJNMJFBLPAAPKGIBB.LLFHBCFKEPCADBBLGKCGDNAEJENGGDBNDNPG() at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.DEIOCOLCLLOAPDPOCJGIJNMJFBLPAAPKGIBB.Run() at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart()


System

- Provider
[ Name] Application Error
- EventID 1000
[ Qualifiers] 0
Level 2
Task 100
Keywords 0x80000000000000
- TimeCreated
[ SystemTime] 2013-06-02T04:00:09.000000000Z
EventRecordID 2294
Channel Application
Computer vc.local
Security

- EventData

VisualCronService.exe
6.1.8.16372
50f40e44
7z64.dll
0.0.0.0
4c7667f6
c0000005
000000000008c350
614
01ce5c6000fe2216
C:\Program Files (x86)\VisualCron\VisualCronService.exe
C:\Program Files (x86)\VisualCron\7z64.dll
ea7a07ff-cb38-11e2-8084-0050568c007f

Thanks for the report. Seems to be some kind of thread/lock issue. We will investigate this. We hope that you find a way to reproduce this. Thanks for the report.

---

Well I cannot reproduce this manually, but it happened again this Wednesday at midnight on schedule. So there is a great likelihood that it will be reproduced this Sunday at Midnight. Do you have any debugging tools or suggestions that we can use to track VisualCron activities at this time for thread/lock related errors.

Thanks,

David

It is not possible to get more debugging than this. But it is probably related to one of these things;

1. the remote server is unavailable in some way
2. the credential used have problems at this particualar moment
3. there are other backup Tasks running in this exact moment, which in some way, causes problems

Any feedback regarding this would help a lot so we can reproduce this.

---

Hi,

we have the same issue on our server VC 7.2.1:

Name der fehlerhaften Anwendung: VisualCronService.exe, Version: 7.2.1.15248, Zeitstempel: 0x541a8993
Name des fehlerhaften Moduls: 7z64.dll, Version: 9.22.0.0, Zeitstempel: 0x4dac8834
Ausnahmecode: 0xc0000005
Fehleroffset: 0x0000000000097370
ID des fehlerhaften Prozesses: 0xc44
Startzeit der fehlerhaften Anwendung: 0x01cff9ca0ee3513d
Pfad der fehlerhaften Anwendung: C:\Program Files (x86)\VisualCron\VisualCronService.exe
Pfad des fehlerhaften Moduls: C:\Program Files (x86)\VisualCron\7z64.dll
Berichtskennung: 104a35bb-665e-11e4-8f19-a0481c953c94

I have just migrate many old command line tasks with zip.exe to the VC archive compress task.
We use a lot of archive compress tasks, therefore we can't live without this....;(

Martin

Hi,

we have the same issue on our server VC 7.2.1:

Name der fehlerhaften Anwendung: VisualCronService.exe, Version: 7.2.1.15248, Zeitstempel: 0x541a8993
Name des fehlerhaften Moduls: 7z64.dll, Version: 9.22.0.0, Zeitstempel: 0x4dac8834
Ausnahmecode: 0xc0000005
Fehleroffset: 0x0000000000097370
ID des fehlerhaften Prozesses: 0xc44
Startzeit der fehlerhaften Anwendung: 0x01cff9ca0ee3513d
Pfad der fehlerhaften Anwendung: C:\Program Files (x86)\VisualCron\VisualCronService.exe
Pfad des fehlerhaften Moduls: C:\Program Files (x86)\VisualCron\7z64.dll
Berichtskennung: 104a35bb-665e-11e4-8f19-a0481c953c94

I have just migrate many old command line tasks with zip.exe to the VC archive compress task.
We use a lot of archive compress tasks, therefore we can't live without this....;(

Martin

It seems like a permission problem. I would advise to upgrade to latest 7.5.0 and see if you can reproduce it. Try to find out which Task. Then send the Task screenshots to support@visualcron.com

---

Hi,

i will update the server and let you know the results.
I dare to exclude a permission problem, because all of our jobs are triggered many, many time each day but the unexpected service stop appear randomly each 1-2 days.

Martin

Hi,

i have updated our VC server to the latest 7.5.0 version, but after 4 days the problem are still present.
It's not possible to reproduce it, because the unexpected service stop appear randomly and seem not to be related to a specific task execution.

We have applied a dirty workaround through a automatic service restart, but this should be only a stopgap solution....

Martin

I see know. It is related to the daily backup or whenever you export settings. Please send screenshots of your settings of that Task to support@visualcron.com

---

Hi,
many thanks for your fast reply.
How already wrote we use a lot of jobs with a lot of archive compress tasks. These jobs are execuded very frequently (many time each hour), therefore are nearly impossible to find out which job cause the service stop.
When i check VC log_server.txt after a unexpected service stop, then the last executed task or job is never the same....

Best Regards,
Martin

Hi,
many thanks for your fast reply.
How already wrote we use a lot of jobs with a lot of archive compress tasks. These jobs are execuded very frequently (many time each hour), therefore are nearly impossible to find out which job cause the service stop.
When i check VC log_server.txt after a unexpected service stop, then the last executed task or job is never the same....

Best Regards,
Martin

As I said it does not seem to be related to Archive Tasks but the backup. So it should be easy to find.

---

Sorry,

but we haven't scheduled backup jobs in our VC configuration.

Martin

Sorry,

but we haven't scheduled backup jobs in our VC configuration.

Martin

Please take a look again. The error is clearly from that Task. And by default we add this "Backup settings" Job. The type of the Task is Export settings Task.

---

Just done,
i confirm that we haven't scheduled "Backup settings" Jobs. We execute the Backup only manually on demand.
Should i search for a particular event in our server logs?

Martin

Just done,
i confirm that we haven't scheduled "Backup settings" Jobs. We execute the Backup only manually on demand.
Should i search for a particular event in our server logs?

Martin

Ok, so this may have happened on demand. Could happen if you do not have the rights to the destination. Please try to reproduce it that way.

---

Hi Henrik,

I can also exclude the on demand setting backup.
Example: During the last weekend we had unexpected service stops on our both VC servers (two redundant servers with nearly the same same settings).
1 unexpected service stop on the first server and 2 unexpected service stops on the second server (and not at the same time).
During the weekend and overnight we are out of the office and therefore nobody start a on demand setting backup...;)

Martin

Hi Henrik,

I can also exclude the on demand setting backup.
Example: During the last weekend we had unexpected service stops on our both VC servers (two redundant servers with nearly the same same settings).
1 unexpected service stop on the first server and 2 unexpected service stops on the second server (and not at the same time).
During the weekend and overnight we are out of the office and therefore nobody start a on demand setting backup...;)

Martin

Ok, but if we can reproduce it by just making a backup it would be so much easier to find the problem. Can you reproduce it?

---