# Building a Web Scraper for Douban Group Posts > Building a Python scraper that downloads featured posts — text and images — from a Douban group into per-post folders: the scraper class, the save logic, the entry-point script, and the scraping problems handled. Source: https://chanmeng.org/blog/douban-group-web-scraper Author: Chan Meng — https://chanmeng.org Published: 2024-10-30 Tags: DevOps --- *Originally published on LinkedIn, 30 October 2024. Republished here with light editing.* ## Introduction In this tutorial, we'll build a Python web scraper that downloads featured posts from [Douban.com](http://Douban.com)'s minimalist lifestyle group. The scraper will save both text content and images from each post into separate folders. ## Prerequisites - Python 3.x - Basic understanding of HTML and web scraping - Required Python packages: requests, beautifulsoup4, hashlib ## Project Structure ``` douban-scraper/ ├── scraper.py # Main scraper class ├── main.py # Script to run the scraper └── posts/ # Downloaded posts (auto-generated) ``` ## Step 1: Creating the Scraper Class First, let's create `scraper.py` with our main scraper class: ```python import requests from bs4 import BeautifulSoup import os import time import re from urllib.parse import urljoin import hashlib class DoubanScraper: def __init__(self): self.headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' } def sanitize_filename(self, filename): # Remove invalid characters and handle long filenames sanitized = re.sub(r'[<>:"/\\|?*]', '', filename) hash_suffix = hashlib.md5(filename.encode()).hexdigest()[:6] if len(sanitized) > 30: sanitized = sanitized[:30] + f"_{hash_suffix}" return sanitized def get_post_content(self, url): response = requests.get(url, headers=self.headers) soup = BeautifulSoup(response.text, 'html.parser') content = soup.find('div', class_='topic-content') if not content: return None, [] images = [] for img in content.find_all('img'): img_url = img.get('src') if img_url: images.append(urljoin(url, img_url)) return content.text.strip(), images ``` ## Step 2: Adding Save Functionality Add the save\_post method to the DoubanScraper class: ```python def save_post(self, title, author, content, images, base_url): folder_name = self.sanitize_filename(title) try: os.makedirs(folder_name, exist_ok=True) image_paths = [] for i, img_url in enumerate(images): img_path = f"{folder_name}/image_{i+1}.jpg" try: img_data = requests.get(img_url, headers=self.headers).content with open(img_path, 'wb') as f: f.write(img_data) image_paths.append(img_path) except Exception as e: print(f"Failed to save image: {e}") continue md_content = f"""# {title} Author: {author} Source: {base_url} ## Content {content} ## Images """ for img_path in image_paths: md_content += f"![Image]({img_path})\n" with open(f"{folder_name}/post.md", 'w', encoding='utf-8') as f: f.write(md_content) print(f"Successfully saved post to folder: {folder_name}") except Exception as e: print(f"Failed to save post: {e}") ``` ## Step 3: Creating the Main Script Create `main.py` to run the scraper: ```python from scraper import DoubanScraper import requests from bs4 import BeautifulSoup import time def main(): # List of posts to skip (optional) skip_titles = ["Post Title 1", "Post Title 2"] scraper = DoubanScraper() base_url = "https://www.douban.com/group/minimalists/discussion?start=0&type=elite" response = requests.get(base_url, headers=scraper.headers) soup = BeautifulSoup(response.text, 'html.parser') table = soup.find('table', class_='olt') if not table: print("Failed to get post list. Login might be required.") return rows = table.find_all('tr')[1:] # Skip header row for row in rows: title_cell = row.find('td', class_='title') if not title_cell: continue link = title_cell.find('a') if not link: continue title = link.text.strip() if any(skip_title in title for skip_title in skip_titles): print(f"Skipping post: {title}") continue url = link['href'] author_cell = row.find('td', attrs={'nowrap': 'nowrap'}) author = author_cell.find('a').text if author_cell and author_cell.find('a') else "Unknown" print(f"Processing: {title} by {author}") content, images = scraper.get_post_content(url) if content: scraper.save_post(title, author, content, images, url) time.sleep(2) # Avoid hitting rate limits if __name__ == "__main__": main() ``` ## Key Features 1. Filename Sanitization: Handles long filenames and removes invalid characters 2. Image Download: Saves all images from posts 3. Markdown Generation: Creates formatted markdown files with post content 4. Rate Limiting: Includes delays to avoid being blocked 5. Skip Functionality: Allows skipping specific posts 6. Error Handling: Robust error handling for network issues ## Challenges Addressed 1. Windows Path Length Limitation: Implemented filename truncation with hash suffixes 2. Anti-Scraping Measures: Added user-agent headers and delays 3. Character Encoding: Proper handling of Chinese characters 4. File Organization: Structured storage of posts and images ## Usage - Install required packages: ```bash pip install requests beautifulsoup4 ``` - Run the scraper: ``` python main.py ``` ## Output Structure For each post, the scraper creates: - A folder named after the post title - A markdown file containing the post content - Downloaded images from the post ## Best Practices 1. Always respect the website's robots.txt 2. Implement reasonable delays between requests 3. Handle errors gracefully 4. Use proper user-agent headers 5. Sanitize filenames for cross-platform compatibility ## Conclusion This scraper demonstrates how to effectively download and organize content from Douban groups while handling common challenges in web scraping. The code is extensible and can be modified for other similar websites. Remember to check the website's terms of service and robots.txt before scraping, and always scrape responsibly! --- Note: This tutorial is for educational purposes only. Always ensure you have permission to scrape content from websites.