View Full Version : Dev-help: Screen Scraping Widget

18th September 2005, 04:23 PM
Hi, I hope there's some Mac developers here that could help me. I would like to write a little widget that will screen-scrape the VirginMobile website. Why? I am a prepaid virgin mobile user and would like to know the credit left on my account, from the dashboard.

But I've hit a snag, I need to be able to authenticate to the virgin mobile website, and it appears to use forms authentication and perhaps use a session cookie to recall who you are while you browse. Let me explain what I've seen while logging in:

1) You navigate to download page
2) You enter your number/password and redirected to authentication page
3) If authenticated to page you redirect to "hello ben" page
4) I now navigate to the page with my account credit details

As we can see between steps 2-4 there's multiple page navigations. And when I try to authenticate to the site using a POST it fails, and I doubt that my authentication (even if it work) would persist.

On the otherhand, if I log into VirginMobile with Safari, and then try my widget-code it will work as the sesion cookies and the webkit (i guess) is authenticated.

*Sigh* is there anyone out there that has some recommendations. I've used curl in the past, but was hoping to use the XMLresponse (i can't remember the name, sorry) object. BTW I threw otu all my code a couple of weeks ago in frustration, so I can't post any code i had, so I am back to concept/design level before venturing further.


20th September 2005, 08:36 AM
I have written a widget that gets images from a smoothwall/ipcop firewall that users authentication, and I found that it is not possible to get the widget to receive authenticated information. I used curl to get the information, and then I referenced the downloaded file.

My guess you best bet would be to have a cocoa backend which does the scraping and passes the information to the frontend.