Packaging with crawl4ai and playwright not possible?

[Rapid1898-code]

Hello - i try to create an executable file for windows out of this code:

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

async def main():
  browser_config = BrowserConfig()
  run_config = CrawlerRunConfig(word_count_threshold=10)
  url = "https://carlsquare.com/de/die-firma/karriere/"
  async with AsyncWebCrawler(config=browser_config) as crawler:
    result = await crawler.arun(
      url=url,      
      config=run_config
    )
  res = result.links
  print(res)
             
if __name__ == "__main__":
    asyncio.run(main())    

using the following command

pyinstaller --onefile --add-data="C:/DEVNEU/.venv/crawl4ai/Lib/site-packages/playwright_stealth/js;./playwright_stealth/js" --collect-data fake_http_header.data --collect-data fake_useragent.data test.py

When i run the python-program everthing works fine - but when i run the created exe-file i get this error:

C:\DEVNEU\Python-Diverses\crawl4ai>test.exe
Traceback (most recent call last):
  File "test.py", line 18, in <module>
    asyncio.run(main())
    ~~~~~~~~~~~^^^^^^^^
  File "asyncio\runners.py", line 194, in run
  File "asyncio\runners.py", line 118, in run
  File "asyncio\base_events.py", line 721, in run_until_complete
  File "test.py", line 9, in main
    async with AsyncWebCrawler(config=browser_config) as crawler:
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "crawl4ai\async_webcrawler.py", line 241, in __aenter__
  File "crawl4ai\async_webcrawler.py", line 225, in start
  File "crawl4ai\async_crawler_strategy.py", line 887, in __aenter__
  File "crawl4ai\async_crawler_strategy.py", line 897, in start
  File "crawl4ai\async_crawler_strategy.py", line 402, in start
  File "playwright\async_api\_generated.py", line 14418, in launch
  File "playwright\_impl\_browser_type.py", line 95, in launch
  File "playwright\_impl\_connection.py", line 61, in send
  File "playwright\_impl\_connection.py", line 528, in wrap_api_call
playwright._impl._errors.Error: BrowserType.launch: Executable doesn't exist at C:\Users\RAPIDT~1\AppData\Local\Temp\_MEI299602\playwright\driver\package\.local-browsers\chromium-1155\chrome-win\chrome.exe
╔════════════════════════════════════════════════════════════╗
║ Looks like Playwright was just installed or updated.       ║
║ Please run the following command to download new browsers: ║
║                                                            ║
║     playwright install                                     ║
║                                                            ║
║ <3 Playwright Team                                         ║
╚════════════════════════════════════════════════════════════╝
[PYI-23236:ERROR] Failed to execute script 'test' due to unhandled exception!
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x0000021C7F49A200>
Traceback (most recent call last):
  File "asyncio\base_subprocess.py", line 130, in __del__
  File "asyncio\base_subprocess.py", line 107, in close
  File "asyncio\proactor_events.py", line 109, in close
  File "asyncio\base_events.py", line 829, in call_soon
  File "asyncio\base_events.py", line 552, in _check_closed
RuntimeError: Event loop is closed

I also tried to run this line where my playwright.exe is stored:

"C:\DEVNEU\.venv\crawl4ai\Scripts\playwright.exe" install

But i still get the same error message.
How is it possible to bundle this module as executable?

[rokm]

Has the browser been downloaded into playwright's package directory (presumably, C:\DEVNEU\.venv\crawl4ai\Lib\site-packages\playwright\driver\package\.local-browsers\chromium-1155)? I think playwright ships a PyInstaller hook that expects that; and while it's known to not be compatible with PyInstaller v6 on linux and macOS, it might work on Windows.

You need to set PLAYWRIGHT_BROWSERS_PATH environment variable, as per https://playwright.dev/python/docs/library#pyinstaller.

[Rapid1898-code]

Thanks for the response - i checked the folder you mentioned - see below

Do you see if the browser is downloaded as its needed?

[Rapid1898-code]

I was now able to bundle this to a working executable -
But the file has over 600MB...
Is there any way to reduce this to a reasonable size at the end?

[rokm]

What is a "reasonable size" for a build that has to bundle one or more web browser builds with it, though?

If you did not install only chrome into your playwright environment but also other browsers, then perhaps you should clean that up and install only the browser(s) your really require (i.e., check what is in site-packages\playwright\driver\package\.local-browsers).

[rokm]

Other than that, it's up to you to determine where any potential extra stuff is coming from, and remove it if possible.

[Rapid1898-code]

Hello - i was able to reduce the size to around 300 MB (still big but better) using the following instuctions and command

https://playwright.dev/python/docs/library#pyinstaller

pyinstaller --onefile --add-data="C:/DEVNEU/.venv/crawl4ai/Lib/site-packages/playwright_stealth/js;./playwright_stealth/js" --collect-data fake_http_header.data --collect-data fake_useragent.data readCrawl.py

If anybody has other suggestions pls get back to me.