Home › Forums › Technical discussion › Configuring Drupal

Importing a big index

Wed, 2008-08-20 06:54 — scottatdrake

A librarian here has created an index of student newspaper articles that go back to the 1900's. The 25,000+ records currently live happily in a MySQL table on our old site (not Drupal).

I was thinking of creating a new content type for the index and simply importing them with the node_import module. This approach worked well for me before when I imported about 2,400 records for something else... but 25,000 is a whole different ball game.

Would you implement a home-grown index with a new content type like this? Are there limits to the number of nodes Drupal can deal with? Would path_auto come to a screeching halt?

‹ Managing users' uploaded files Managing open hours ›
Wed, 2008-08-20 08:19 — Cary Gordon

Test it

This appears to be one of those questions that is best answered by testing. 25k is not a lot of records or nodes, per se, but it is hard to predict potential pitfalls that one might encounter in a specific Drupal configuration. Since this is not so big that it wouldn't run on almost anything, why not copy your site onto a dev server - your laptop, perhaps - and give it a try. Even if it fails, it might give you some insight as to what workaround is required.

Cary Gordon
The Cherry Hill Company

Wed, 2008-08-20 11:52 — scottatdrake

Well, that's sensible

Well, that's sensible enough. Test it.

Regarding the notion that 25,000 nodes is not a lot of records... Sounds like I underestimate Drupal's ability to scale. Thanks, Cary.

Mon, 2008-08-25 13:34 — Anonymous (not verified)

25,000 records shouldn't be a problem

25,000 shouldn't be a problem for Drupal to handle. However - you may not want to import them all at the same time. Speaking from previous experience, depending on your php configuration settings, the node import script may time out.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <h3> <h4> <h5> <h6> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote>
  • Lines and paragraphs break automatically.
  • You may post code using <code>...</code> (generic) or <?php ... ?> (highlighted PHP) tags.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Use [toc list: ol; title: Table of Contents; minlevel: 2; maxlevel: 3; attachments: yes;] to insert a mediawiki style collapsible table of contents. All the arguments are optional.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
          _       ___   ____     ___      _ 
   __ _  | |     |_ _| | __ )   / _ \    (_)
  / _` | | |      | |  |  _ \  | (_) |   | |
 | (_| | | |___   | |  | |_) |  \__, |   | |
  \__,_| |_____| |___| |____/     /_/   _/ |
                                       |__/
Enter the code depicted in ASCII art style.