Problem:
I need to download many files like these

http://www.site.com/page.php?id=101
http://www.site.com/page.php?id=102
http://www.site.com/page.php?id=103

the page is the same, the content is dinamic, depending on the id passed.

I don’t want to download any unnecessary software to do a simple task like this, so I search a little…

Preparation:

I wrote all the links in a text file, one per line. (A list is always usefull)
Let’s say the file is filelist.txt

Solution for Windows 7:

I can use the built-in Powershell !!
There is the link in Start->All Programs->Accessories->Windows Powershell->Windows Powershell
Btw, I managed to reach the powershell guessing I had to type “powershell” in the console.

First I need to import the module for the Background Intelligent Transfer Service (BITS) with:

PS C:> Import-Module bitstransfer

Then I must provide the list of files formatted as this:

Source, Destination
http://www.url.of/your/source.file, c:\path\where\to\save\my_file_with.extension
http://www.url.of/your/source2.file, c:\path\where\to\save\other_file.extension

(Remember that “Source, Destination” is the actual first line of filelist.txt)

Once I have the file formatted correctly I can use

PS C:> Import-CSV filelist.txt | Start-BitsTransfer

to read the input file and download all the file.. Good!

…
NOT GOOD! Bits doesn’t support dynamic content!

BITS supports HTTP and HTTPS downloads and uploads and requires that the server supports the HTTP/1.1 protocol. For downloads, the HTTP server’s Head method must return the file size and its Get method must support the Content-Range and Content-Length headers. As a result, BITS only transfers static file content and generates an error if you try to transfer dynamic content, unless the ASP, ISAPI, or CGI script supports the Content-Range and Content-Length headers.

I couldn’t download the files with a batch, I just made an html file with many links pointing to the URLs and downloaded by hand (Save link as..).

Solution for Linux:

Open a console
Type “wget -i filelist.txt”

Anything to say?