https://bogza.ro/index.php?action=history&feed=atom&title=Using_AliExpress_Scraper_And_Update_The_Wiki_Pages 2026-09-15T12:38:35Z Revision history for this page on the wiki MediaWiki 1.45.4 https://bogza.ro/index.php?title=Using_AliExpress_Scraper_And_Update_The_Wiki_Pages&diff=1600&oldid=prev 2025-10-06T18:20:31Z

<p>Automation proof-of-concept update</p> <p><b>New page</b></p><div>== Description ==<br /> Using AliExpress Scraper and Updating Wiki Pages<br /> <br /> == Prerequisites ==<br /> * Python 3.10 or newer<br /> * Google Chrome installed (for Selenium rendering)<br /> * ``pip install -r requirements.txt``<br /> * ``playwright install chromium``<br /> <br /> == Scraping a Single Product ==<br /> === Run the scraper with the AliExpress product URL ===<br /> ``python aliexpress_scraper.py &quot;https://www.aliexpress.com/item/PRODUCT_ID.html&quot; --output extracted_items``<br /> <br /> === Key options ===<br /> * ``--show-browser`` – launch Chrome and Playwright with visible windows (solve captchas by hand).<br /> * ``--output`` – where to store the extracted item folder (defaults to the current directory).<br /> <br /> === Environment helpers ===<br /> * ``CHROME_USER_DATA_DIR`` – path to a dedicated Chrome profile folder for automation.<br /> * ``CHROME_PROFILE_DIRECTORY`` – profile name (for example ``Default``) when reusing an existing profile.<br /> * ``CHROME_BINARY_PATH`` – explicit path to ``chrome.exe`` if Chrome is not on the default path.<br /> <br /> === Outputs are written to ``&lt;output&gt;/&lt;Sanitised Product Title&gt;/`` with ===<br /> * ``images/`` – converted JPEG product images.<br /> * ``description_images/`` – assets found inside the description block.<br /> * ``summary.json`` – structured metadata (title, URL, description, product_id, extracted_text).<br /> * ``info.txt`` – human-readable summary.<br /> * ``page.html`` and ``dom.txt`` – raw and rendered HTML snapshots.<br /> <br /> == Scraping Multiple URLs and Updating Wiki Pages ==<br /> === Use the pipeline helper when you want to scrape and publish in one go ===<br /> ``python scrape_and_publish.py &lt;url1&gt; &lt;url2&gt; --base extracted_items``<br /> <br /> === Options ===<br /> * ``--skip-wiki`` – only scrape.<br /> * ``--dry-run`` – perform uploads in dry-run mode (no MediaWiki changes).<br /> * ``--base`` – reuse an existing extraction folder.<br /> <br /> == Publishing Existing Folders to MediaWiki ==<br /> === Configure the following environment variables (``.env`` file supported) ===<br /> * ``MEDIAWIKI_API_URL`` – e.g. ``https://wiki.example.org/w/api.php``<br /> * ``MEDIAWIKI_BOT_USERNAME`` / ``MEDIAWIKI_BOT_PASSWORD`` – bot credentials.<br /> * ``MEDIAWIKI_PAGE_PREFIX`` – optional namespace or prefix for generated pages.<br /> * ``MEDIAWIKI_USER_AGENT`` – contact string for your automation client.<br /> * ``MEDIAWIKI_EDIT_SUMMARY`` – edit summary used for page saves.<br /> <br /> === Then run ===<br /> ``python create_wiki_pages.py extracted_items``<br /> <br /> === The script will ===<br /> # Load configuration from the environment (or ``.env``).<br /> # Collect each subfolder under the base directory.<br /> # Upload any ``images/`` and ``description_images/`` files (skipping duplicates).<br /> # Format ``description.txt`` into wiki markup and create/update pages.<br /> <br /> == Workflow Tips ==<br /> * Keep ``extracted_items/`` under version control ignore (already added to ``.gitignore``).<br /> * Use ``--show-browser`` if AliExpress challenges you with captchas; the scraper waits for you to solve them.<br /> * Regenerate wiki pages after editing ``description.txt`` by re-running ``create_wiki_pages.py``.<br /> * For auditing, inspect ``summary.json`` and ``info.txt`` before publishing.<br /> * Combine ``scrape_and_publish.py`` with a text file of URLs for batch imports.</div>

TinelBot