You can even save a data extraction configuration files, to be used in new project, or elsewhere.
I've been using kind of Xpath for years with php.
and you don't need to start with it : Start with smart, or with wizard, and then Edit in Advanced Mode. But of course, the Advanced Mode is the most important part. Sometimes you need to find alternate ones. hidden behind an 'Display' Ajax button that I wasn't able to deal with (with php / cUrl) 10 tasks are offered for free, and as far I know, won't be public tasks as it's the case with some of Octoparse competitors Smart Mode and Wizard mode make it easy to find the data, often at first sight. because I was unable to access the most important part of the data I needed. as if it wouldn't be any ajax routines on the pages. Several reasons for it : easy to set up lots of tutorials to start easily Ajax is handled as easy as a basic html url. had to be fast, had to be robust ! I gave a try to some scraping tools, and my final choice was made to Octoparse. In two word : a nightmare ! So, I had to find a way to still be able to extract my needed data, without having to pass an engineer degree in information technology. and the dynamic pages that don't load at first sight, that wait for you to click on a button, that just show as you scroll down, that exchange static pictures urls with javascipt dynamically shown pictures. Then came for me (and I must admit, my limited skills) THE hammer : AJAX ! Yes, html + Javascipt + css + dom. In fact, websites regularly change minor things on their pages, and in the best case, you wouldn't get anymore some or all of the awaited data, in the worse case, absolutely inaccurate data. Years after years, it sounded clear that my extracting routines running on my server were more and more difficult to maintain in a good working shape. I have been crawling and parsing websites for a while, with use of php and cUrl. I wish I had discovered this jewel years ago.
0 kommentar(er)
