Welcome to WebmasterWorld Guest from

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

How do I automate raw logs download

Vertex control panel and daily log...

12:38 pm on Oct 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 20, 2003
votes: 0

I manually click on my vertex control panel "download raw logs" each day.

I wonder how to automate that process (I have 2 sites and each day I must log in and click each link)

Any script? Tool?

I know I can use the server statistics but that's not the point, I also need that files on my pc..

1:32 pm on Oct 15, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Jan 13, 2002
votes: 0

How about FTP?
5:49 pm on Oct 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 20, 2003
votes: 0

and how do I schedule that?
the file has a date format like mysite-10-10-2003.gz ...
so everyday has a different name..
6:12 pm on Oct 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 16, 2001
votes: 0

There's probably a more elegant way to do it. But since I'm not a programmer, this is what I would do:

1) Create a batch FTP job that retrieves all your .gz files to a local directory.
2) Decompress it (gzip -d).
3) Then use xcopy /y (for DOS) which suppresses prompting to confirm you want to overwrite a file.

If search long enough, you'll find command line tools that you can also use batch rename files or perform some processing for you. Its all very cool what you can find already written by geeks out there.

1:31 pm on Oct 16, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 28, 2002
votes: 0

You need two programs one to build the file to get the logs the other to actually get the logs. This is what I use, it may not be the most elegant or fast but it does work.

use the scheduler to run them, run build_log_gatherer.vbs first then get_yourdomain.bat

I run this

program 1 - build_log_gatherer.vbs

option explicit

'this program gets the log files from an FTP Server

dim build_filename' the name of the file that will need to run from the scheduler
dim build_directory' the location it will be built
dim login_filename' the name of the file with the login info
dim ftp_file' the file we want to get
dim dest_loc' final location of the file that will be ftp'd
dim ftp_site' location to ftp to
dim ftp_directory' ftp directory
dim ftp_username' login username
dim ftp_password' login password
dim temp_str, temp_strII

build_filename= "get_files.bat"
build_directory= "c:/logs/"
login_filename= "get_yourdomain.txt"
ftp_file= ""
dest_loc= "c:/logs/"
ftp_site= "" ' put ftp site here
ftp_directory= ""' put ftp logs here
ftp_username= ""' put username here
ftp_password= ""'put password here
temp_str= ""
temp_strII= ""

call write_blank(build_directory, login_filename)
call write_send(build_directory, login_filename, ftp_username)
call write_send(build_directory, login_filename, ftp_password)
call write_send(build_directory, login_filename, "cd " & ftp_directory)
call write_send(build_directory, login_filename, "get " & log_filename(ftp_file))

call write_blank(build_directory, build_filename)
call write_send(build_directory, build_filename, "cd " & dest_loc)
call write_send(build_directory, build_filename, "ftp -i -s:" & build_directory & login_filename & " " & ftp_site)
call write_send(build_directory, build_filename, "bye " )

function write_send(loc, file, txt)
Dim ofs
Dim objtextstream
Set ofs = CreateObject("Scripting.FileSystemObject")
Set objtextstream = ofs.openTextFile(loc & file,8, True)
objtextstream.writeline txt
Set objtextstream = Nothing
Set ofs = Nothing
end function

function write_blank(loc, file)
dim fso, MyFile
Set fso = CreateObject("Scripting.FileSystemObject")
Set MyFile = fso.CreateTextFile(loc & file, True)
end function

function log_filename(ftp_file)
'put the prefix for your files in the first set of quotes in the next line
ftp_file = "" & left(monthname(month(now()-1)), 3)
if day(now()-1) < 10 then ftp_file = ftp_file & "0"
ftp_file = ftp_file & day(now()-1) & "_" & year(now())
log_filename = ftp_file
end function

program 2 get_yourdomain.bat

cd c:/logs/
ftp -i -s:c:/logs/get_yourdomain.txt yourdomain.com