i trying script wget command download web page , it's attachments , jpegs etc.
when enter script hand, works, need run on 35000 times archive old web site outside of control (international company politics, i'm owner of data).
my problem has been in variablising session parameters.
my script far follows:
cnt=35209 # initialise headers general_settings='-4 -p xyz --restrict-file-names=windows -nc --limit-rate=250k' html_page_specific='--convert-links --html-extension' proxy='--proxy-user=xxxxxx --proxy-password=yyyyyyy' session="--header=\'host: mywebsite.com:9090\' --header=\'user-agent: mozilla/5.0 (windows nt 5.1; rv:20.0) gecko/20100101 firefox/20.0\'" address=http://mywebsite.com:9090/browse/item-$cnt echo $general_settings $proxy $session $cookie $address echo echo echo getting item-$cnt... #while [ $cnt -gt 0 ] #do # # page wget --debug $general_settings $html_page_specific $proxy $session $cookie $address # attachments, pdf, txt, jpg, gif, sql, etc... # wget -a.pdf $general_settings -r $proxy $session $cookie $address # wget -a.txt $general_settings -r $proxy $session $cookie $address # wget -a.jpg $general_settings -r $proxy $session $cookie $address # wget -a.gif $general_settings -r $proxy $session $cookie $address # wget -a.sql $general_settings -r $proxy $session $cookie $address # wget -a.doc $general_settings -r $proxy $session $cookie $address # wget -a.docx $general_settings -r $proxy $session $cookie $address # wget -a.xls $general_settings -r $proxy $session $cookie $address # wget -a.xlsm $general_settings -r $proxy $session $cookie $address # wget -a.xlsx $general_settings -r $proxy $session $cookie $address # wget -a.xml $general_settings -r $proxy $session $cookie $address # wget -a.ppt $general_settings -r $proxy $session $cookie $address # wget -a.pptx $general_settings -r $proxy $session $cookie $address # wget -a.png $general_settings -r $proxy $session $cookie $address # wget -a.ps $general_settings -r $proxy $session $cookie $address # wget -a.mdb $general_settings -r $proxy $session $cookie $address # ((cnt=cnt-1)) # #done but when run script, following output
getting item-35209... setting --inet4-only (inet4only) 1 setting --directory-prefix (dirprefix) xyz setting --restrict-file-names (restrictfilenames) windows setting --no (noclobber) 1 setting --limit-rate (limitrate) 250k setting --convert-links (convertlinks) 1 setting --html-extension (htmlextension) 1 setting --proxy-user (proxyuser) xxxxx setting --proxy-password (proxypassword) yyyyy setting --header (header) \'host: setting --header (header) 'cookie: debug output created wget 1.11.4 red hat modified on linux-gnu. as can see, host , cookie sections not being formatted, resulting in wget command failing log in , extract data.
i've been reading bash man pages, googling, , have tried several related suggestions so, i'm still unable command execute.
anyone out there going nice enough show me correct way quote quotes in veriables?
thanks,
quotes inside of quoted strings or variables ordinary characters, not quoting characters. there's no way change that. use array instead:
a=(a b 'c d' 'e f') cmd "${a[@]}" calls cmd 4 arguments a, b, c d, , e f.
(you achieve similar effect eval, that's lot more error prone. in case, using arrays more convenient.)
Comments
Post a Comment