i'm running php web scraper grabs info online stores (disclaimer, have permission stores this). scraper has 3 functions:
- gets list of store urls json file (for ease, let's i'm passing 1 url in function though)
- function
getlinksgrabs of links on supplied url page, finds "next item" link, goes it, grabs product links page, , on. puts them in array$alllinks. - i run
foreach ($alllinks $key => $value) {getinfo($value);} - function
getinfo($value)goes each product url , gets product information each individual product page.
here full code http://pastebin.com/shhbg9en#
on server causing errors because (afaik) i'm using memory/the code quite cumbersome. can print out results of getlinks fine, scrape deeper 500 server error (and nothing posted log whatsoever).
my question
should writing results mysql , lighten load of program? if so, what's best way it? need (somehow) clear variables or free memory? i'm new php intricacies of optimising code foreign me.
you should enable error_reporting
error_reporting(e_all); ini_set("display_errors","on"); i think should consider multi curl see
Comments
Post a Comment