web scraping - Optimising a PHP web scraper to avoid 500 server errors -


i'm running php web scraper grabs info online stores (disclaimer, have permission stores this). scraper has 3 functions:

  1. gets list of store urls json file (for ease, let's i'm passing 1 url in function though)
  2. function getlinks grabs of links on supplied url page, finds "next item" link, goes it, grabs product links page, , on. puts them in array $alllinks.
  3. i run foreach ($alllinks $key => $value) {getinfo($value);}
  4. function getinfo($value) goes each product url , gets product information each individual product page.

here full code http://pastebin.com/shhbg9en#

on server causing errors because (afaik) i'm using memory/the code quite cumbersome. can print out results of getlinks fine, scrape deeper 500 server error (and nothing posted log whatsoever).

my question

should writing results mysql , lighten load of program? if so, what's best way it? need (somehow) clear variables or free memory? i'm new php intricacies of optimising code foreign me.

you should enable error_reporting

error_reporting(e_all); ini_set("display_errors","on");   

i think should consider multi curl see


Comments