Using AliExpress Scraper And Update The Wiki Pages
Description
Using AliExpress Scraper and Updating Wiki Pages
Prerequisites
- Python 3.10 or newer
- Google Chrome installed (for Selenium rendering)
- ``pip install -r requirements.txt``
- ``playwright install chromium``
Scraping a Single Product
Run the scraper with the AliExpress product URL
``python aliexpress_scraper.py "https://www.aliexpress.com/item/PRODUCT_ID.html" --output extracted_items``
Key options
- ``--show-browser`` – launch Chrome and Playwright with visible windows (solve captchas by hand).
- ``--output`` – where to store the extracted item folder (defaults to the current directory).
Environment helpers
- ``CHROME_USER_DATA_DIR`` – path to a dedicated Chrome profile folder for automation.
- ``CHROME_PROFILE_DIRECTORY`` – profile name (for example ``Default``) when reusing an existing profile.
- ``CHROME_BINARY_PATH`` – explicit path to ``chrome.exe`` if Chrome is not on the default path.
Outputs are written to ``<output>/<Sanitised Product Title>/`` with
- ``images/`` – converted JPEG product images.
- ``description_images/`` – assets found inside the description block.
- ``summary.json`` – structured metadata (title, URL, description, product_id, extracted_text).
- ``info.txt`` – human-readable summary.
- ``page.html`` and ``dom.txt`` – raw and rendered HTML snapshots.
Scraping Multiple URLs and Updating Wiki Pages
Use the pipeline helper when you want to scrape and publish in one go
``python scrape_and_publish.py <url1> <url2> --base extracted_items``
Options
- ``--skip-wiki`` – only scrape.
- ``--dry-run`` – perform uploads in dry-run mode (no MediaWiki changes).
- ``--base`` – reuse an existing extraction folder.
Publishing Existing Folders to MediaWiki
Configure the following environment variables (``.env`` file supported)
- ``MEDIAWIKI_API_URL`` – e.g. ``https://wiki.example.org/w/api.php``
- ``MEDIAWIKI_BOT_USERNAME`` / ``MEDIAWIKI_BOT_PASSWORD`` – bot credentials.
- ``MEDIAWIKI_PAGE_PREFIX`` – optional namespace or prefix for generated pages.
- ``MEDIAWIKI_USER_AGENT`` – contact string for your automation client.
- ``MEDIAWIKI_EDIT_SUMMARY`` – edit summary used for page saves.
Then run
``python create_wiki_pages.py extracted_items``
The script will
- Load configuration from the environment (or ``.env``).
- Collect each subfolder under the base directory.
- Upload any ``images/`` and ``description_images/`` files (skipping duplicates).
- Format ``description.txt`` into wiki markup and create/update pages.
Workflow Tips
- Keep ``extracted_items/`` under version control ignore (already added to ``.gitignore``).
- Use ``--show-browser`` if AliExpress challenges you with captchas; the scraper waits for you to solve them.
- Regenerate wiki pages after editing ``description.txt`` by re-running ``create_wiki_pages.py``.
- For auditing, inspect ``summary.json`` and ``info.txt`` before publishing.
- Combine ``scrape_and_publish.py`` with a text file of URLs for batch imports.