i'm running php web scraper grabs info online stores (disclaimer, have permission stores this). scraper has 3 functions:
- gets list of store urls json file (for ease, let's i'm passing 1 url in function though)
- function
getlinks
grabs of links on supplied url page, finds "next item" link, goes it, grabs product links page, , on. puts them in array$alllinks
. - i run
foreach ($alllinks $key => $value) {getinfo($value);}
- function
getinfo($value)
goes each product url , gets product information each individual product page.
here full code http://pastebin.com/shhbg9en#
on server causing errors because (afaik) i'm using memory/the code quite cumbersome. can print out results of getlinks
fine, scrape deeper 500 server error (and nothing posted log whatsoever).
my question
should writing results mysql , lighten load of program? if so, what's best way it? need (somehow) clear variables or free memory? i'm new php intricacies of optimising code foreign me.
you should enable error_reporting
error_reporting(e_all); ini_set("display_errors","on");
i think should consider multi curl see
Comments
Post a Comment