I am currently keeping an eye on properties in Lyon, France. The process has been tedious and called out for some automation. Megan and I plan to return to France in the future and this little project should ease the burden of finding an apartment or house.

Croix-Paquet, Lyon

This morning I discovered the perl module Web::Scraper. It is a port of a Ruby based tool called scrAPI. The approach taken avoids regular expression matching and opts for XPath and DOM tree selector matching; both more resilient methods of addressing specific sections of a web page.

Apartments, Lyon

I found one stumbling block that took a while to overcome. After a little trial and error, I discovered the FireFox browser returned misleading XPaths for objects embedded in tables.

The XPaths provided by FireBug and XPather, included browser-inserted tbody tags. These tags did not appear in my source web pages. Thus the browser’s XPath did not match the structure used by Web::Scraper, and caused Web::Scraper to miss the desired content.

The solution was easy; strip out the tbody tags and Web::Scraper returns to working as advertised.

With this problem overcome, the project is already looking helpful.