Thomas Hardy Poems

[Declanworld]

Evening all.

There are 947 Hardy poems available to download individually on this page .  Anyone have any idea if it's possible to get the whole heap in one go?

[Buffer6]

Title: Poems of the Past and the Present


Author: Thomas Hardy

You could try https://www.gutenberg.org/ebooks/3168https://www.gutenberg.org/ebooks/3168 

[Declanworld]

Thanks comrade. That's a great option but it's only one collection. I've nearly come to the realisation that my page can't harvested in one go.

[asim18]

Yes it's possible.

[Declanworld]

Okay, shoot.

[asim18]

Basically you need to generate a list of URLs first. Luckily the URLs are easy to generate because it's a simple range from 0000000001.doc to 0000000948.doc.

Once you have the list of 948 URLs saved as .txt, you need to run a script to download all the URLs in the list. Would recommend adding a 2 second sleep in the script so as to not overload the server with a thousand sudden HTTP requests.

[goslow]

---

@Declanworld wrote:

Evening all.

There are 947 Hardy poems available to download individually on this page.  Anyone have any idea if it's possible to get the whole heap in one go?

---

I have just tried this Chrome extension, 'Simple Mass Downloader'

https://chrome.google.com/webstore/detail/simple-mass-downloader/abdkkegmcbiomijcbdaodaflgehfffed

on a prototype machine and it seems to do what you want. Downloaded the first page of 100 doc's on page 1 very fast. You'd have to repeat for x10 pages.

The files on the site though are named with just numbers so you end up with 100 files numbered 1 to 100 in your download folder! It will also grab the first 5 'best known' files at the top of each page unless you deselect them each time to avoid duplicates/overwriting.

You have to turn off Chrome's feature to prompt for the download folder destination (which is off by default) otherwise it will prompt for each file 100 times.

Have only just Googled and tried this extension and nothing more but seems to do what you want. Use at your own risk!

You then might need another utility to rename all the file numbers!

(Edit: There does appear to be a renaming feature in the advanced features of the extension.)

[Declanworld]

I'm blown away with the stupendous service from asim18 and goslow.

Minutes after using asim18's superb extension (Firefox version) to download all the poems, goslow weighs in with a zip file of the whole lot.  Now I have them twice!

Thanks so very much to you both for your knowledge, time and kind consideration.

PS: That Simple Mass Downloader will undoubtedly be very useful in the future.