Menu
Reply
Declanworld
  • 260
  • 2
  • 19
Superfast
1,069 Views
Message 1 of 8
Flag for a moderator

Thomas Hardy Poems

Evening all.

There are 947 Hardy poems available to download individually on this pageAnyone have any idea if it's possible to get the whole heap in one go?

0 Kudos
Reply
Buffer6
  • 12.3K
  • 94
  • 654
Hero
883 Views
Message 2 of 8
Flag for a moderator

Re: Thomas Hardy Poems

Title: Poems of the Past and the Present


Author: Thomas Hardy

You could try https://www.gutenberg.org/ebooks/3168https://www.gutenberg.org/ebooks/3168 

Declanworld
  • 260
  • 2
  • 19
Superfast
868 Views
Message 3 of 8
Flag for a moderator

Re: Thomas Hardy Poems

Thanks comrade. That's a great option but it's only one collection. I've nearly come to the realisation that my page can't harvested in one go.
0 Kudos
Reply
asim18
  • 278
  • 12
  • 89
Fibre optic
853 Views
Message 4 of 8
Flag for a moderator

Re: Thomas Hardy Poems

Yes it's possible.

Declanworld
  • 260
  • 2
  • 19
Superfast
841 Views
Message 5 of 8
Flag for a moderator

Re: Thomas Hardy Poems

Okay, shoot.
0 Kudos
Reply
asim18
  • 278
  • 12
  • 89
Fibre optic
832 Views
Message 6 of 8
Flag for a moderator

Re: Thomas Hardy Poems

Basically you need to generate a list of URLs first. Luckily the URLs are easy to generate because it's a simple range from 0000000001.doc to 0000000948.doc.

Once you have the list of 948 URLs saved as .txt, you need to run a script to download all the URLs in the list. Would recommend adding a 2 second sleep in the script so as to not overload the server with a thousand sudden HTTP requests.

goslow
  • 2.07K
  • 332
  • 714
Problem sorter
767 Views
Message 7 of 8
Flag for a moderator

Re: Thomas Hardy Poems


@Declanworld wrote:

Evening all.

There are 947 Hardy poems available to download individually on this pageAnyone have any idea if it's possible to get the whole heap in one go?


I have just tried this Chrome extension, 'Simple Mass Downloader'

https://chrome.google.com/webstore/detail/simple-mass-downloader/abdkkegmcbiomijcbdaodaflgehfffed

on a prototype machine and it seems to do what you want. Downloaded the first page of 100 doc's on page 1 very fast. You'd have to repeat for x10 pages.

The files on the site though are named with just numbers so you end up with 100 files numbered 1 to 100 in your download folder! It will also grab the first 5 'best known' files at the top of each page unless you deselect them each time to avoid duplicates/overwriting.

You have to turn off Chrome's feature to prompt for the download folder destination (which is off by default) otherwise it will prompt for each file 100 times.

Have only just Googled and tried this extension and nothing more but seems to do what you want. Use at your own risk!

You then might need another utility to rename all the file numbers!

(Edit: There does appear to be a renaming feature in the advanced features of the extension.)

Declanworld
  • 260
  • 2
  • 19
Superfast
735 Views
Message 8 of 8
Flag for a moderator

Re: Thomas Hardy Poems

I'm blown away with the stupendous service from asim18 and goslow.

Minutes after using asim18's superb extension (Firefox version) to download all the poems, goslow weighs in with a zip file of the whole lot.  Now I have them twice!

Thanks so very much to you both for your knowledge, time and kind consideration.

PS: That Simple Mass Downloader will undoubtedly be very useful in the future.