Scheduled Access Log Downloads

How to automate the renaming of log file for each day

[geckofuel]

I'm trying to automate the process of downloading my access log files each day, while saving the log file with a unique name for each day.

What I'd like is to get this:

widgets-020903.org
widgets-021003.org
widgets-021103.org
etc.

However, on my remote server, the log file is simply saved as:

widgets.org

Every scheduling FTP program I use simply downloads it with the same name (widgets.org), overwriting the previous day's file.

Any suggestions?

[lorax]

What platform are you using for your workstation?

[bill]

If you're on Windows you could probably write a simple batch file that would use the command-line version of FTP to access your log file and then rename it with the current date after transferring it to your local machine. Then schedule the batch file to run daily.

[andreasfriedrich]

Use wget [wget.org]. wget will not overwrite an existing file. It will add a numeric suffic´x to the file instead. Depending on your skills and/or needs you might want to rename the file later on using the right date.

Andreas

[geckofuel]

I'm working on a machine with Windows 2000.

[lorax]

Windows 2000 comes with a Scheduler program which in combination with a batch file (written in DOS script) should accomplish what you want.

[microsoft.com...]

[aspdaddy]

My daily logs are just named by date -
eg yesterdays is "ex030211.log"

Would it be possible in DOS script/batch files to create the filename dynamically from the system date, open the dos ftp program , connect and download to my c:/ drive, and do this automatically every day?

Thanks

[lorax]

Yes aspdaddy it is possible. I was playing with that idea a few weeks ago. I canned the idea after I bought NetTracker as it did what I needed automatically. Check your FTP program - it may have scripting ability built into the GUI.

But here's where left off with what I was working on:

a file called something.cmd

ftp -v -i -g -s:d:\somedir\params.txt

date/t¦cut -f 2 -d " "¦sed s/\([0-9]\{2\}\).\([0-9]\{2\}\).\([0-9]\{4\}\)/"ren access_log.1 \3-\1-\2_log"/g > c:\winnt\today.cmd
date/t¦cut -f 2 -d " "¦sed s/\([0-9]\{2\}\).\([0-9]\{2\}\).\([0-9]\{4\}\)/"ren error_log \3-\1-\2_errorlog"/g >> c:\winnt\today.cmd
c:\winnt\today.cmd

params.txt is:

open www.yourdomain.com
yourftpuname
yourftppwd
lcd d:\pathtodownload\logfilesto\
get logs/error_log
get logs/access_log.1
bye

The first file calls two Unix commands that a coworker created/found for the Windows environment. SED.exe and cut.exe. SED allows regex and cut - well that's self explanatory. Stickyme your email address and I'll send them along to you.

[geckofuel]

I've written up a batch file / ftp script combo that performs a download of the access logs, renaming them with a datestamp.

*yourdomainlog.bat* (in your bat file, include only the text inside the stars)

*************************
:: yourdomainlog.bat
@ECHO off

Set CURRDATE=%TEMP%\CURRDATE.TMP
Set CURRTIME=%TEMP%\CURRTIME.TMP

DATE /T > %CURRDATE%
TIME /T > %CURRTIME%

Set PARSEARG="eol=; tokens=1,2,3,4* delims=/, "
For /F %PARSEARG% %%i in (%CURRDATE%) Do SET YYYYMMDD=%%l%%k%%j

Set PARSEARG="eol=; tokens=1,2,3* delims=:, "
For /F %PARSEARG% %%i in (%CURRTIME%) Do Set HHMM=%%i%%j%%k

ftp -s:yourdomainftp.txt ftp.yourdomain.org>>yourlogfile_%YYYYMMDD%.log
:END

*************************

*yourdomainftp.txt* include only the text inside the stars

*************************
yourusername

yourpassword

get yourlogfile

close

bye
*************************

After you have your yourdomainlog.bat and yourdomainftp.txt files, put them in the directory that you want to download your log files to. Then, in Windows 2000 (not sure about other operating systems) set up a a Task Schedule to run yourdomainlog.bat everyday or every week or whatever.

I'd be happy to help anyone who needs clarification.

[edited by: geckofuel at 5:10 pm (utc) on Feb. 13, 2003]

[lorax]

Similar idea geckofuel but doesn't require the extra tools mine does. Elegantly done. :)