Automate downloading ~400 files with semi-consistent names ?

[michaelsanford]

I have to download a huge batch of files from a web site named as:
• first file, first part = 001a.txt
• third file, second part = 003b.txt
etc ... up to
• 114.txt

The only problem is some files have only one part and some go up to f ; the files with only one part have no letter tag. I don't want to have to copy-paste each individual file name into a transfer manager, the list just too voluminous.

I don't have FTP access and the web folder where the files are kept is not allowed directory listing.

I ideally want to download the files one at a time, automatically with a ferw terminal commands (or a GUI app, whatever). Each file is about 15 megs and more than one file at a time isn't efficient on my connection.
Can I do this with curl, wget, or any other fink-able package ? Any and all ideas welcome...

Luckily, each file is listed on a single page with links and on that page every link with an extension ".txt" is a file I need. I'm thinking a script with sed might be useful. I was thinking that I could strip the HTML source file of everything except a CR/tab deliminated list of each file I need and then somehow pass that to curl or something ? I don't know enough shell scripting to be able to do that though...

 

[michaelsanford]

Ok, I've half-answered my own question. I've managed to use sed, grep and awk to reduce the page's source to single lines of the format "http://blah.blah.com/001a.txt" quotes and all (I can't figure out how to remove them with sed).

My problem now is that curl -O `cat fixed.txt | grep 01 | grep 10` returns
curl: (1) Unsupported protocol: "http
curl: (1) Unsupported protocol: "http

(I use the two greps to only return two items in the list, as a test, but eventually there won't be any).

I suppose I have to strip the quotation marks from the lines, but how do I do that with sed ?

EDIT:
The examples I used for sed were all of the format sed "s///" which made the " a terminating character; I just didn't wrap the command with quotes and it worked just fine.