Here is the original Presentation given on Crawlers, from early in the semester.  Some points of consideration for the papers presented herein might include:

Heydon & Najork, "Mercator:  A Scalable, Extensible Web Crawler"

• Single-threaded vs. multi-threaded crawling:  How does this decision influence ease-of-coding & bottleneck elimination?
  
• Content-seen vs. URL-seen tests:  How are they done, and what is the distinction?
  
• Scalability & extensibility
  

Pant, Srinivasan, Menczer, "Crawling the Web"

• Comprehensiveness vs. preferential crawling:  When are these used, and why?
  
• Crawling vs. searching:  Different, though functionally inter-related.
  
• How can we avoid spider traps?
  

     (Questions taken from p.  23 of the paper, as they seem highly pertinent:)

• Can crawlers help a search engine to focus on user interests?
• Can a search engine help a crawler to focus on a topic?
• Can a crawler on one machine help a crawler on another?

This discussion on crawling dynamic pages is also very relevant.