Problem:
I need to download many files like these
http://www.site.com/page.php?id=101
http://www.site.com/page.php?id=102
http://www.site.com/page.php?id=103
the page is the same, the content is dinamic, depending on the id passed.
I don’t want to download any unnecessary software to do a simple task like this, so I search a little…
Preparation:
I wrote all the links in a text file, one per line. (A list is always usefull)
Let’s say the file is filelist.txt
Solution for Windows 7:
I can use the built-in Powershell !!
There is the link in Start->All Programs->Accessories->Windows Powershell->Windows Powershell
Btw, I managed to reach the powershell guessing I had to type “powershell” in the console.
First I need to import the module for the Background Intelligent Transfer Service (BITS) with:
PS C:> Import-Module bitstransfer
Then I must provide the list of files formatted as this:
Source, Destination
http://www.url.of/your/source.file, c:\path\where\to\save\my_file_with.extension
http://www.url.of/your/source2.file, c:\path\where\to\save\other_file.extension
(Remember that “Source, Destination” is the actual first line of filelist.txt)
Once I have the file formatted correctly I can use
PS C:> Import-CSV filelist.txt | Start-BitsTransfer
to read the input file and download all the file.. Good!
…
NOT GOOD! Bits doesn’t support dynamic content!
BITS supports HTTP and HTTPS downloads and uploads and requires that the server supports the HTTP/1.1 protocol. For downloads, the HTTP server’s Head method must return the file size and its Get method must support the Content-Range and Content-Length headers. As a result, BITS only transfers static file content and generates an error if you try to transfer dynamic content, unless the ASP, ISAPI, or CGI script supports the Content-Range and Content-Length headers.
I couldn’t download the files with a batch, I just made an html file with many links pointing to the URLs and downloaded by hand (Save link as..).
Solution for Linux:
Open a console
Type “wget -i filelist.txt”
Anything to say?












