[Solved] FTP Monitor Folder then Download and Email New/Amended Files - VisualCron - Forum

Community forum

FCStone
2013-06-27T10:11:33Z
I'm new to VisualCron and been given the task of being the tester to decide if this nice little package will become our automation client.

Trying to get a task setup that monitors an FTP folder and triggers tasks when new files or amended files are added to this FTP folder. I can only copy from this folder, I cannot move, add or delete. This part I have got working.

Then the first task is to download the file(s) from the FTP into a temporary folder, and then email the newly downloaded files to a specific distribution list.

The part I'm having trouble with is the attachments on the email. I only want to email the files that the previous task downloaded. Now simply attaching everything in a folder isn't working as it'll email older files already sent. But if I delete the older files so the Email task won't send them twice, then the Download task will just re-download the deleted files and they are emailed anyways.

I was hoping some kind of variable could be passed from the Download task to the Email task, but so far nothing is working.

Any help would be appreciated.
Support
2013-06-27T13:33:58Z
What if you process one file at a time? By using the Remote file trigger you can use the Variables as in the screenshot below.

When a new file has arrived you use the Variables to download and then later in the email attachment. Please let us know if you need any further help.

Support attached the following image(s):

Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-06-27T13:50:41Z
I tried something like that, but the variables only ever returned the details of the first file of the group of files that caused the trigger.

But I have solved it.

I have a folder that retains all files downloaded, and the Download task only overwrites if newer. So only newly added or updated files are downloaded by this task.

This task then passes its DownloadedFilesNames variable to a loop containing a Copy task and runs this task for every filename in that variable. The Copy task copies all the files specified by the variable to another temporary folder. Then the Email task launches, attaches all files in that second temp folder and then a Delete task clears the second temp folder.

Therefore the email can contain multiple files, but only newly downloaded ones.
FCStone
2013-06-28T16:45:11Z
A new issue with this task I'm afraid. I'm having trouble getting time exceptions to work correctly. I have two versions of this job, running side by side to try and figure out what the issue was. Only difference is one has the time exception and the other doesn't. The time exception is to not allow the job to run from 12am to 5pm. But the job with the time exception isn't triggering at all while the version without it is triggering fine. And the job trigger just fine is receiving triggers after 5pm.

Any help on this?
Support
2013-06-28T23:38:29Z
Have you looked at the realtime log when this is happening? Please post screenshots of the Time exceptions.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-01T08:12:10Z
The log for Friday for that job was blank, I've attached a screenshot of the exception as best as I can as well as a screenshot of my two parallel jobs, totally identical except the exception. The bottom one is working, the top one is not.

FCStone attached the following image(s):
Support
2013-07-01T12:07:08Z
I think the problem is that the way the Time exception and Remote file trigger is designed. What happens is that the Remote file trigger always are looking for new files. When a new file has arrived the Trigger is being fired - if a a time exception is not met. At that time the file is already marked as found and new and will never Trigger again (with this file).

I think that is the reason it is not working for you.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-01T12:11:53Z
I did think that, but the parallel job had successful triggers after 5pm when the time exception should have stopped preventing triggers.
Support
2013-07-01T12:26:55Z
The logs should help you in this case. Please turn on Extended debug logging in server settings. Then look at the log when you expect it to trigger. Remember that they may not (the Jobs) trigger at the exact same time depending on trigger interval and when they were created.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-01T12:42:14Z
But Friday's log of the job with the exception was blank, it never triggered. Yet the other job was receiving triggers as late as 5:20pm; twenty minutes after the exception stopped preventing triggers. Not sure what the log is going to tell me if it remains blank.
Support
2013-07-01T12:51:57Z
Which file do you look at?
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-01T12:54:54Z
I'm afraid you've lost me with that last question. What do you mean?
Support
2013-07-01T12:55:38Z
I mean the log file.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-01T13:14:13Z
I'm just right clicking on the job and selecting "Job Log". IS that not the correct log?
Support
2013-07-01T13:15:27Z
By default, it has not very long history. You should look at the file log_serverDATE.txt from the log directory of installation folder.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-01T13:37:26Z
The plot thickens, the log file is showing that the exception was still blocking the triggers as late as 5:20, but the exception is set for 12am to 5pm as shown by my earlier screenshot. Any ideas?
Support
2013-07-01T13:39:57Z
Yes, when you set 5PM you will also block all the minutes during the whole 5PM.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-01T13:41:03Z
And now I hug you. Excellent. Changing that now, lets see how it runs today. Thankyou
Support
2013-07-01T13:43:03Z
😁 Let me know how it goes!
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-02T17:07:58Z
New problems I'm afraid. The Copy loop I'm using relies upon the DownloadedFileNames variable from the previous task, which is a FTP Download task. But this variable seems to be temperamental.

On today's run, even though it downloaded 10 files, the DownloadedFileNames variable only contained the name of the file with the latest modified date as shown on the FTP, and thus that is all the Copy task moved across for emailing. Yesterday it worked fine.

I've looked through the log, and all files were downloaded in a single run of the Download task, the file contained in the variable was the 2nd file to be downloaded in the task, so not even a special position like first or last.

This task now has two triggers after what you said yesterday about exceptions. It triggers on the FTP event, but also has an initial timed trigger that happens the moment the exception period is over at 5:15pm. Both triggers were in place yesterday and it worked fine.

I think this issue is somehow linked to the FTP Trigger event, as the filename in the variable would have been the last file to trigger it, but the exception would have prevented task activation.

Any thoughts?
Support
2013-07-02T21:44:18Z
Originally Posted by: FCStone 

New problems I'm afraid. The Copy loop I'm using relies upon the DownloadedFileNames variable from the previous task, which is a FTP Download task. But this variable seems to be temperamental.

On today's run, even though it downloaded 10 files, the DownloadedFileNames variable only contained the name of the file with the latest modified date as shown on the FTP, and thus that is all the Copy task moved across for emailing. Yesterday it worked fine.

I've looked through the log, and all files were downloaded in a single run of the Download task, the file contained in the variable was the 2nd file to be downloaded in the task, so not even a special position like first or last.

This task now has two triggers after what you said yesterday about exceptions. It triggers on the FTP event, but also has an initial timed trigger that happens the moment the exception period is over at 5:15pm. Both triggers were in place yesterday and it worked fine.

I think this issue is somehow linked to the FTP Trigger event, as the filename in the variable would have been the last file to trigger it, but the exception would have prevented task activation.

Any thoughts?



I am not sure how the FTP Trigger event is related? Werent you using Time Triggers?


Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-03T05:28:40Z
I'm using both, an FTP Trigger and a Time Trigger. The exception stops the job running at 5:15pm where upon at 5:15pm an initial Time Trigger goes off to catch all files that were uploaded during the exception period. Then from 5:15pm to 12am, the job relies upon the FTP Trigger in case any amended or additional files are uploaded after that point.

But I don't know how the FTP Trigger is related, I'm just trying to figure out why the DownloadedFileNames variable is only containing 1 file name.
FCStone
2013-07-03T16:23:28Z
Okay today's run of this job was even worse than yesterdays, the DownloadedFileNames variable returned blank after the Download task downloaded 10 files. I'm at a loss why this variable is not working. The time period to focus on is 5:15pm when the initial trigger occurs.

My deadline on this benchmark of your software is this Friday, if I can't get this job working without issue by then, I'll have to source an alternative solution. Ideally I don't want to do this because I believe this package is what my company needs. I'm only saying this because Support hasn't posted any further help on this thread today.
FCStone attached the following image(s):
Support
2013-07-03T19:38:42Z
What is important to know is that this DownloadFileNames are dynamic to the specific runtime. I think the 2 Triggers have been setup in a way that prevents this from working.

We have a lot of questions. For example how do you know it is more than one name?

I suggest creating a File write Task before the Task with the look so you can see what is in the Variable. Add a time stamp as well so you know when it was run.

You could also add some more screenshots of the full setup.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
FCStone
2013-07-03T21:01:43Z
I understand you'll need all the information to help the best you can, so I've done as many screenshots as I think are viable to help give you a decent picture.

Also I did something similar to your suggestion in that it writes the variable to the body of the email. When yesterday when it downloaded 10 files in one go, it only copied the one file for emailing, and only that files name was in the body, meaning it was the only one in the variable, where as it should of been all 10 file names divided by carriage returns from what I've seen through testing.
File Attachment(s):
Screenshots.zip (669kb) downloaded 38 time(s).
Scroll to Top