Community forum

Please note that VisualCron support is not actively monitoring this community forum. Please use our contact page for contacting the VisualCron support directly.


stevja01
2014-03-12T18:55:09Z
Our business owner has asked us to do the following and we can't see a way to do this in VisualCron.

We've written a .bat script that will retain a rolling 52 week average file size for a given retailer file that we receive weekly. If the current week's file is +/- 15% from this average we need to send an email to the client.

Where can we save and update this 52 week average file size so that we can send an alert if our current file size is +/- 15%?
Sponsor
Forum information
Support
2014-03-12T21:07:24Z
The .NET Code Condition seems to be the best option for this. Create a Job, schedule it daily. Add an email Task. Create a .NET Code condition, write the code for verifying this. Check the Condition on the Job. Now, you will only get the email when it is a match.
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
stevja01
2014-03-12T21:18:57Z
Thanks Henrik, .NET is a bit out of our league so we don't know how to go about doing that. We'll read up on it and see what we can do. Thanks for the info.
bbusse
2014-03-18T01:05:11Z
Originally Posted by: stevja01 

Thanks Henrik, .NET is a bit out of our league so we don't know how to go about doing that. We'll read up on it and see what we can do. Thanks for the info.



Does that .bat file contain any proprietary information or is that something you'd be able to share? I'm quite sure we can come up with something, but i'd need to know what your batch file is doing and how the results look. My initial thoughts are that your batch file would write it's average file size out to a file that contains nothing but the average size of the file as its contents. Visualcron reads that file to get the value, does a calculate (script task with some code) to figure out the +/- 15%, and if it returns a certain value, trigger an e-mail task (or any other task).

EDIT: I didn't read Henrik's post properly. I completely agree with his suggestion. Only trick is to get the value your batch file says is the average written somewhere, and then use the condition as he suggested.

Brian
Kevin Rood
2014-03-18T14:12:21Z
Thanks for the reply Brian. You're exactly right.

Our batch file doesn't contain anything proprietary. Our data is zipped, so the batch file simply gets the last 52 weeks folders, and runs a 'for' loop inside each folder that:
- unzips the contents
- reads each of the files and sizes
- records the filename and size into a text (csv) file
- deletes the files just unzipped, leaving only the original

Then, once all file sizes are recorded...
- Reads through the resulting csv and calculates the average file size
- Calculates the upper and lower thresholds (+/- 15%)
- Writes the threshold results to two individual files

So, in short, I've already got two separate files that only contain the upper and lower file size thresholds respectively. From there, I'd just like VisualCron to:
- IF the new file is within the threshold, copy it to folder location 1 (where other jobs will begin based on a file trigger).
- IF the new file is outside the threshold, copy it to location 2 (so that we can specify the names of the 'bad' files in an email alert to the client).

I was hoping that the VisualCron "Copy files" task where the file sizes are specified could accept the {FILE(Content} variable instead of keying in a fixed file size threshold, but that's not possible. But that's the crux of the issue we've got.

Any ideas based on that information?
Thanks again for the help.
bbusse
2014-03-18T14:22:09Z
I might just be tired (i am haha), but i'm having trouble writing this out the way I want... but i'll try.

So you have a new file today, lets say:
File: File1.zip
Size: 40000kb

You have 2 files that contain something like

FileLowContents: 10000kb
FileHighContents: 50000kb


So you'd want to read both of those files, see that the size of file1.zip is 40000kb and therefor between those ranges, and go ahead and copy to location one.

If not between those 2 sizes (lets say it was 60000kb), it would copy it elsewhere and generate an e-mail.

Is that accurate?

If so, can you upload those 2 resulting files so I see how they look?

Brian
Kevin Rood
2014-03-18T14:46:46Z
Sure thing.
Here are two files that are generated by the .bat

Two files attached:
PSCNZO6407_MAX.info
PSCNZO6407_MIN.info

The file we'd get weekly from the client would be something like PSCNZO6407-20140318.DETAIL, and your description of what I'm trying to do is 100% accurate.

P.S. The numbers in each of those files is in KB.
File Attachment(s):
PSCNZO6407_MAX.txt (1kb) downloaded 53 time(s).
PSCNZO6407_MIN.txt (1kb) downloaded 47 time(s).
bbusse
2014-03-18T14:48:59Z
Perfect. Thanks. The Size in the MAX file, for example is 310769. Is that in bytes or kbytes?

EDIT: Saw your edit :)

Brian
Kevin Rood
2014-03-18T14:50:43Z
You beat me to it.
Edited my post to say Kilobytes, and by the time I saved, you had replied!

But yes, kilobytes.

Thanks!!
bbusse
2014-03-18T16:28:05Z
Ok, I've tried every way I know how to do this without it getting EXTREMELY convoluted.

I think what we have here is a potential feature request instead of a solution.

Henrik, Based on a few different methods I was trying I think the following ideas might be good to implement, if possible.


1. On the CopyFiles SIZE criteria, instead of forcing a numeric value being typed or raised/lowered by the up/down arrows, give an option to specify a variable. This would have to potentially see a string value (numbers) and convert it to INT. I tried setting a variable to the output value of a read file task, which did read the numbers from his MIN/MAX files, but they were still strings and therefore I couldn't do much with it using any of the built-in stuff expecting an Int32 value.

2. Currently there is a List File(s) task, I think it would be beneficial to have a "Get File Attributes" task that could be used against a single file, especially within a loop. So we could do a List Files task, get the file name (or names), pass each one through a loop which would use a "Get File Attributes" task to get the size, or the date.. or whatever attribute (one attribute at a time to keep it simple). One task to get the date, One task to get the size, etc. We could then set variables to the output of these tasks and do all sorts of stuff :)

2. On a File Condition, we currently only have the option to see that it exists (true/false) or that the content matches/contains/etc.. a string you can input. It would be beneficial if other file attributes could be selected as criteria also. Such as:
File Size Greater Than (numeric choice or variable input):
File Size Less than (numeric choice or variable input):
File Date is greater/less/equal to 'x' whether that be a variable or selected number of days/weeks/etc old.

Basically, the same type of things we already have available when doing a Copy File(s) task, only use those same potential criteria for an exact file.... in a Condition.


Now, as for Kevin's request. I don't honestly see a way to cleanly do 'exactly' what Kevin requested and I described a few posts ago. I can do it in PowerShell (put code in powershell task) to determine the min/max and whether it's within the threshold, but native support would be ideal.. even if it takes several tasks in a loop to get there.

Since you need a solution, I'll provide this.. 🙂 It does work, and it's actually quite clean. It just requires powershell (language i'm comfortable with)


1 Job, 3 tasks.

Task 1: Process-->PowerShell
Code contents, for this example:

$File = gci C:\Temp\VisualCron\PSCNZO6407-20140318.DETAIL
[int]$Min = get-content C:\Temp\VisualCron\PSCNZO6407_MIN.txt
[int]$Max = get-content C:\Temp\VisualCron\PSCNZO6407_MAX.txt
[int]$Size = $file.length / 1kb

if ($Size -lt $Max -and $Size -gt $Min)
{return $True}
else
{return $False}

It will get the details of the PSCNZ06407-20140318.DETAIL file (can modify that to be dynamic so it's repeatable daily), read the contents of your 2 min/max files, does some math to determine that it is within the min/max (True) or not (false), and returns that true or false as the Output of the task.

Task 2: Copy File Task: Name it something like "Copy FILENAME - GOOD SIZE"
Simply configure it to copy the file where it should go, if it's a proper size.

Task 3: Same as above, only copy the file to the 'BAD' location and name it appropriately.

Now, for how this works. It's all dependant on the powershell task FLOW.

So, edit your powershell task and go to the FLOW tab. I have 3 items
1. On Complete, if Output equal "True" - Goto Task "Copy FILENAME - GOOD SIZE".
2. On Complete, if Output equal "False" - Goto Task "Copy FILENAME - BAD SIZE".
3. On Error, Stop Job

So, if your tasks are in this order:
Powershell
Copy Good
Copy Bad
........... Edit the Copy Good task and change it's FLOW tab to stop the job regardless. So on mine, I have 2 entries under FLOW.
1. On Success, Stop the Job
2. On Error, Stop the Job.

The reason for this is that The Powershell task got true... and launched this task. We do NOT want the NEXT task to run, because that's the Copy BAD task. So we quit now. Any tasks that run after the Copy Good task, will not run, which is desired.

Now, validate that your 'Copy Bad' task has a FLOW like the following:
On Success, continue with Next Task
On Error, Stop Job

Now you can add another task to send an e-mail also or you can make it a notification for 'Copy Bad'. Just add another 'On Complete' flow item that launches a notification. Either way works. If using the e-mail task, the only time it'll run is if the PowerShell task was 'False' since any other result ended in the job stopping. :)

Hopefully that makes sense. I'll attach screenshots detailing the 3 tasks (no e-mail).
















Brian
bbusse attached the following image(s):
bbusse
2014-03-18T16:30:45Z
Only 6 attachments per post.. here's the 7th.
bbusse attached the following image(s):
Kevin Rood
2014-03-18T17:05:19Z
Wow. Working in software and this being my first post on the forum, I have to say that all companies should strive for this level of support. Kudos.


One last question on the variable filename in Powershell...Is there any way to translate this VisualCron variable:
{DATEFORMAT(yyyy)}-WK-{MATH(Subtract|Integer|{DATE(Week)}|1|#000)}
into Powershell?

We get many of our files in that format. In the above example, it'd be something like PSCNZO6407.2014-WK-011.DETAIL

And I'm not a Powershell expert to know if/how it'd be possible to get that date format as a variable in PS.

Thanks again for your help. If there is a place I can give a positive review of the support, please let me know.

Kevin
bbusse
2014-03-18T17:35:48Z
Originally Posted by: Kevin Rood 

Wow. Working in software and this being my first post on the forum, I have to say that all companies should strive for this level of support. Kudos.



You're very welcome. I should note, that I do not work for VisualCron. I just happen to use the software quite extensively since 2007 or so for the large bank I work for. Henrik was kind enough to make me an admin so I can do more on the forum. Thx again for that, btw :)

Originally Posted by: Kevin Rood 


One last question on the variable filename in Powershell...Is there any way to translate this VisualCron variable:
{DATEFORMAT(yyyy)}-WK-{MATH(Subtract|Integer|{DATE(Week)}|1|#000)}
into Powershell?

We get many of our files in that format. In the above example, it'd be something like PSCNZO6407.2014-WK-011.DETAIL

And I'm not a Powershell expert to know if/how it'd be possible to get that date format as a variable in PS.



You're in luck. I'm not sure that it'd work if you had a powershell script (.ps1) file saved, as I haven't tried, but if you can use the PowerShell task with the code embedded (like my example) the variables definitely work.

Simply replace the first line with this: (anything in the brackets from what i'm aware of, will be translated directly)

$File = gci C:\Temp\VisualCron\PSCNZO6407.{DATEFORMAT(yyyy)}-WK-{MATH(Subtract|Integer|{DATE(Week)}|1|#000)}.DETAIL



Originally Posted by: Kevin Rood 


Thanks again for your help. If there is a place I can give a positive review of the support, please let me know.

Kevin



Anytime. I use the forums to further my knowledge. You wouldn't believe the things you can learn trying to come up with solutions for others 🙂 So as such, thanks for the questions :)

(If you just want to click 'Thank' on the post, that's about all i'm aware of you can do in these forums)

Brian
Kevin Rood
2014-03-18T18:14:21Z
Originally Posted by: bbusse 


You're in luck. I'm not sure that it'd work if you had a powershell script (.ps1) file saved, as I haven't tried, but if you can use the PowerShell task with the code embedded (like my example) the variables definitely work.



This is great news.
I will definitely need to get more familiar with Powershell, knowing that, and now realizing I can combine it into multiple outputs with $True/$False.

Many thanks again.

Kevin
bbusse
2014-03-18T19:01:51Z
No problem at all. Happy to help.

And Just one more FYI. I chose to do the 'return $true' and 'return $false' as an example. You can use the 'return' option in your script (last thing you would do) and anything after return is exactly what the output would be for the task. It could've been like this:

return "Good"

or

return "Bad"

and the output would've just plain said Good or Bad instead of True/False. I just like True/False as it's easy to keep generic.

Brian
Support
2014-03-18T19:26:31Z
Originally Posted by: bbusse 

Ok, I've tried every way I know how to do this without it getting EXTREMELY convoluted.

I think what we have here is a potential feature request instead of a solution.

Henrik, Based on a few different methods I was trying I think the following ideas might be good to implement, if possible.


1. On the CopyFiles SIZE criteria, instead of forcing a numeric value being typed or raised/lowered by the up/down arrows, give an option to specify a variable. This would have to potentially see a string value (numbers) and convert it to INT. I tried setting a variable to the output value of a read file task, which did read the numbers from his MIN/MAX files, but they were still strings and therefore I couldn't do much with it using any of the built-in stuff expecting an Int32 value.

2. Currently there is a List File(s) task, I think it would be beneficial to have a "Get File Attributes" task that could be used against a single file, especially within a loop. So we could do a List Files task, get the file name (or names), pass each one through a loop which would use a "Get File Attributes" task to get the size, or the date.. or whatever attribute (one attribute at a time to keep it simple). One task to get the date, One task to get the size, etc. We could then set variables to the output of these tasks and do all sorts of stuff :)

3. On a File Condition, we currently only have the option to see that it exists (true/false) or that the content matches/contains/etc.. a string you can input. It would be beneficial if other file attributes could be selected as criteria also. Such as:
File Size Greater Than (numeric choice or variable input):
File Size Less than (numeric choice or variable input):
File Date is greater/less/equal to 'x' whether that be a variable or selected number of days/weeks/etc old.

Basically, the same type of things we already have available when doing a Copy File(s) task, only use those same potential criteria for an exact file.... in a Condition.



Brian



1. I have taken notes about that. Seems like a good request to me

2. instead you could use the Variables for this. We have added a lot of File Variables for retrieving size, date etc. even with Variables.

3. instead maybe you can use the Variable Condition and combine it with File Variables
Support attached the following image(s):

Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
bbusse
2014-03-18T19:31:56Z
Originally Posted by: Support 


1. I have taken notes about that. Seems like a good request to me

2. instead you could use the Variables for this. We have added a lot of File Variables for retrieving size, date etc. even with Variables.

3. instead maybe you can use the Variable Condition and combine it with File Variables



Ok, I have to apologize. I didn't even notice the File Variables had been updated to include so many options. Last I saw there were 3 options for File Content and File.GetRandomFileName/File.GetTempFileName. That's absolutely great. I've been working with 6.1.4 for so long that I completely missed that update and didn't look on my 7.1.1 or 7.1.4 installations we just rolled out.

Brian
Kevin Rood
2014-03-18T20:22:15Z
Originally Posted by: Support 

3. instead maybe you can use the Variable Condition and combine it with File Variables



This is also a winner!

VisualCron is awesome!

:)

Thanks guys!!
Support
2014-03-19T08:54:01Z
Thanks for the feedback!

bbusse - I do not blame you for not following all changes within VisualCron - I am having a hard time myself to track all changes!

Kevin - hope we got a solution for you. Thanks!
Henrik
Support
http://www.visualcron.com 
Please like  VisualCron on facebook!
Scroll to Top