Building a Web Scraper for Douban Group Posts

Originally published on LinkedIn, 30 October 2024. Republished here with light editing.

Introduction#

In this tutorial, we'll build a Python web scraper that downloads featured posts from Douban.com's minimalist lifestyle group. The scraper will save both text content and images from each post into separate folders.

Prerequisites#

Python 3.x
Basic understanding of HTML and web scraping
Required Python packages: requests, beautifulsoup4, hashlib

Project Structure#

douban-scraper/
├── scraper.py     # Main scraper class
├── main.py        # Script to run the scraper
└── posts/         # Downloaded posts (auto-generated)

Step 1: Creating the Scraper Class#

First, let's create scraper.py with our main scraper class:

import requests
from bs4 import BeautifulSoup
import os
import time
import re
from urllib.parse import urljoin
import hashlib
 
class DoubanScraper:
    def __init__(self):
        self.headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
        }
 
    def sanitize_filename(self, filename):
        # Remove invalid characters and handle long filenames
        sanitized = re.sub(r'[<>:"/\\|?*]', '', filename)
        hash_suffix = hashlib.md5(filename.encode()).hexdigest()[:6]
        if len(sanitized) > 30:
            sanitized = sanitized[:30] + f"_{hash_suffix}"
        return sanitized
 
    def get_post_content(self, url):
        response = requests.get(url, headers=self.headers)
        soup = BeautifulSoup(response.text, 'html.parser')
 
        content = soup.find('div', class_='topic-content')
        if not content:
            return None, []
 
        images = []
        for img in content.find_all('img'):
            img_url = img.get('src')
            if img_url:
                images.append(urljoin(url, img_url))
 
        return content.text.strip(), images

Step 2: Adding Save Functionality#

Add the save_post method to the DoubanScraper class:

def save_post(self, title, author, content, images, base_url):
    folder_name = self.sanitize_filename(title)
 
    try:
        os.makedirs(folder_name, exist_ok=True)
 
        image_paths = []
        for i, img_url in enumerate(images):
            img_path = f"{folder_name}/image_{i+1}.jpg"
            try:
                img_data = requests.get(img_url, headers=self.headers).content
                with open(img_path, 'wb') as f:
                    f.write(img_data)
                image_paths.append(img_path)
            except Exception as e:
                print(f"Failed to save image: {e}")
                continue
 
        md_content = f"""# {title}
 
Author: {author}
Source: {base_url}
 
## Content
{content}
 
## Images
"""
 
        for img_path in image_paths:
            md_content += f"![Image]({img_path})\n"
 
        with open(f"{folder_name}/post.md", 'w', encoding='utf-8') as f:
            f.write(md_content)
 
        print(f"Successfully saved post to folder: {folder_name}")
 
    except Exception as e:
        print(f"Failed to save post: {e}")

Step 3: Creating the Main Script#

Create main.py to run the scraper:

from scraper import DoubanScraper
import requests
from bs4 import BeautifulSoup
import time
 
def main():
    # List of posts to skip (optional)
    skip_titles = ["Post Title 1", "Post Title 2"]
 
    scraper = DoubanScraper()
    base_url = "https://www.douban.com/group/minimalists/discussion?start=0&type=elite"
 
    response = requests.get(base_url, headers=scraper.headers)
    soup = BeautifulSoup(response.text, 'html.parser')
 
    table = soup.find('table', class_='olt')
    if not table:
        print("Failed to get post list. Login might be required.")
        return
 
    rows = table.find_all('tr')[1:]  # Skip header row
 
    for row in rows:
        title_cell = row.find('td', class_='title')
        if not title_cell:
            continue
 
        link = title_cell.find('a')
        if not link:
            continue
 
        title = link.text.strip()
 
        if any(skip_title in title for skip_title in skip_titles):
            print(f"Skipping post: {title}")
            continue
 
        url = link['href']
 
        author_cell = row.find('td', attrs={'nowrap': 'nowrap'})
        author = author_cell.find('a').text if author_cell and author_cell.find('a') else "Unknown"
 
        print(f"Processing: {title} by {author}")
 
        content, images = scraper.get_post_content(url)
        if content:
            scraper.save_post(title, author, content, images, url)
 
        time.sleep(2)  # Avoid hitting rate limits
 
if __name__ == "__main__":
    main()

Key Features#

Filename Sanitization: Handles long filenames and removes invalid characters
Image Download: Saves all images from posts
Markdown Generation: Creates formatted markdown files with post content
Rate Limiting: Includes delays to avoid being blocked
Skip Functionality: Allows skipping specific posts
Error Handling: Robust error handling for network issues

Challenges Addressed#

Windows Path Length Limitation: Implemented filename truncation with hash suffixes
Anti-Scraping Measures: Added user-agent headers and delays
Character Encoding: Proper handling of Chinese characters
File Organization: Structured storage of posts and images

Usage#

Install required packages:

pip install requests beautifulsoup4

Run the scraper:

python main.py

Output Structure#

For each post, the scraper creates:

A folder named after the post title
A markdown file containing the post content
Downloaded images from the post

Best Practices#

Always respect the website's robots.txt
Implement reasonable delays between requests
Handle errors gracefully
Use proper user-agent headers
Sanitize filenames for cross-platform compatibility

Conclusion#

This scraper demonstrates how to effectively download and organize content from Douban groups while handling common challenges in web scraping. The code is extensible and can be modified for other similar websites.

Remember to check the website's terms of service and robots.txt before scraping, and always scrape responsibly!

Note: This tutorial is for educational purposes only. Always ensure you have permission to scrape content from websites.