In data mining, much time is spent on collecting raw data and preprocessing data. Quite a few data mining research tasks require to download data from Internet, e.g. Wikipedia articles from Wikipedia, photos from Flickr, Google search results, reviews from Amazon, etc.
Usually general purpose crawlers, such as wget, are not sufficiently powerful and specialized in downloading data from Internet. Writing a crawler on our own is often required.
Recently I am doing an image related research project, in which I need to download a lot of tagged images from Flickr. I am aware that there is a Flickr downloadr, which uses Flickr API to download images. However 1) it only downloads licensed photos and 2) it cannot download the tags of a photo. Thus I decided to write one myself.
The input of the is a tag query, e.g. “dog”, # of photos to download and the disk folder to store the downloaded images.
Because the number of photos is quite big, so downloading them in parallel is critical. In .Net, there are several ways to do parallel computing. For IO intensive tasks, Async workflow is the best.
Tutorials for Async workflow
As Async workflow is one of the key features of F#, there are quite a few tutorials online for F# Async programming. Providing one more in this blog would be repetious.
Luke Hoban’s PDC 2009 talk, F# for Parallel and Asynchronous Programming, is very good for beginners.
Don Syme wrote 3 articles in a series. Highly recommended for experienced F# users!
The Flickr crawler
To write a crawler for a web site like Flickr, we need to 1) design the downloading strategy and 2) analyze the structures of Flickr.
My strategy is to use the search query to search images with some specific tags and from the result page(as shown below), the url of each image is extracted, from which the image and its tags are then crawled.
Figure 1. Flickr search result page.
Figure 2. Flickr image page with an image and its tags.
So first we need a function to download a web page asynchronously :
fetchUrl pretends to be a Mozilla browser and can do some redirections if the url is slightly invalid. The current exception handling is very easy – just return the empty string for the web page. Notice that the return type of the function is Async<string>, thus it cannot be used to download images as images are of binary format, not text.
So the next task is to write a function to download images:
Next we write the code to get the url of an image and its tags from an image page(see Figure 2):
Finally, write a function to work through every search result page, parse the result page and download the images in that result page:
with a driver function to write all the images into hard disk:
We’ve done! A Flickr image crawler in only about 120 lines of code. Let’s download some images!
downloadImagesWithTag "sheep" 5 @"D:\WORK\ImageData\flickr\sheep\"
It only costs less than 5 minutes to download 300 sheep pictures from Flickr.
1. One of the strengths of F# Async programming is its ease for exception handling. In this example, the exception is handled immediately, we can also propagate the exception into an upper level function and handle it there. However, that would require more thinking….
2. I only parallelly download images in one search page. The program could be modified to parallelly process multiple search result pages, which is done sequentially now. If done in this way, we can see that, we can build a hierarchical parallel program: 1) at the first level, multiple search result pages are processed parallelly and 2) at the second level, images in a research result pages are downloaded parallelly.