Using AliExpress Scraper And Update The Wiki Pages

From B-Wiki
Jump to navigation Jump to search

Description

Using AliExpress Scraper and Updating Wiki Pages

Prerequisites

  • Python 3.10 or newer
  • Google Chrome installed (for Selenium rendering)
  • ``pip install -r requirements.txt``
  • ``playwright install chromium``

Scraping a Single Product

Run the scraper with the AliExpress product URL

``python aliexpress_scraper.py "https://www.aliexpress.com/item/PRODUCT_ID.html" --output extracted_items``

Key options

  • ``--show-browser`` – launch Chrome and Playwright with visible windows (solve captchas by hand).
  • ``--output`` – where to store the extracted item folder (defaults to the current directory).

Environment helpers

  • ``CHROME_USER_DATA_DIR`` – path to a dedicated Chrome profile folder for automation.
  • ``CHROME_PROFILE_DIRECTORY`` – profile name (for example ``Default``) when reusing an existing profile.
  • ``CHROME_BINARY_PATH`` – explicit path to ``chrome.exe`` if Chrome is not on the default path.

Outputs are written to ``<output>/<Sanitised Product Title>/`` with

  • ``images/`` – converted JPEG product images.
  • ``description_images/`` – assets found inside the description block.
  • ``summary.json`` – structured metadata (title, URL, description, product_id, extracted_text).
  • ``info.txt`` – human-readable summary.
  • ``page.html`` and ``dom.txt`` – raw and rendered HTML snapshots.

Scraping Multiple URLs and Updating Wiki Pages

Use the pipeline helper when you want to scrape and publish in one go

``python scrape_and_publish.py <url1> <url2> --base extracted_items``

Options

  • ``--skip-wiki`` – only scrape.
  • ``--dry-run`` – perform uploads in dry-run mode (no MediaWiki changes).
  • ``--base`` – reuse an existing extraction folder.

Publishing Existing Folders to MediaWiki

Configure the following environment variables (``.env`` file supported)

  • ``MEDIAWIKI_API_URL`` – e.g. ``https://wiki.example.org/w/api.php``
  • ``MEDIAWIKI_BOT_USERNAME`` / ``MEDIAWIKI_BOT_PASSWORD`` – bot credentials.
  • ``MEDIAWIKI_PAGE_PREFIX`` – optional namespace or prefix for generated pages.
  • ``MEDIAWIKI_USER_AGENT`` – contact string for your automation client.
  • ``MEDIAWIKI_EDIT_SUMMARY`` – edit summary used for page saves.

Then run

``python create_wiki_pages.py extracted_items``

The script will

  1. Load configuration from the environment (or ``.env``).
  2. Collect each subfolder under the base directory.
  3. Upload any ``images/`` and ``description_images/`` files (skipping duplicates).
  4. Format ``description.txt`` into wiki markup and create/update pages.

Workflow Tips

  • Keep ``extracted_items/`` under version control ignore (already added to ``.gitignore``).
  • Use ``--show-browser`` if AliExpress challenges you with captchas; the scraper waits for you to solve them.
  • Regenerate wiki pages after editing ``description.txt`` by re-running ``create_wiki_pages.py``.
  • For auditing, inspect ``summary.json`` and ``info.txt`` before publishing.
  • Combine ``scrape_and_publish.py`` with a text file of URLs for batch imports.
Retrieved from "https://bogza.ro/index.php?title=Using_AliExpress_Scraper_And_Update_The_Wiki_Pages&oldid=1600"