Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google no longer supports javascript-free search requests #1211

Open
2 of 8 tasks
buzzert opened this issue Jan 16, 2025 · 82 comments
Open
2 of 8 tasks

Google no longer supports javascript-free search requests #1211

buzzert opened this issue Jan 16, 2025 · 82 comments
Labels
bug Something isn't working unfortunate Something is very unfortunate

Comments

@buzzert
Copy link

buzzert commented Jan 16, 2025

Describe the bug
As of late, I'm starting to get empty search results when searching for something.

Screenshot:
image

It actually appears the HTML returned from Whoogle contains a fragment of an error message from Google (I presume):

<html lang="en"><head><title>Google Search</title><style>body{background-color:#fff}</style></head><body><noscript><style>table,div,span,p{display:none}</style><meta content="0;url=/httpservice/retry/enablejs?sei=Z3KIZ57GL8_O0PEPgcvWoQc" http-equiv="refresh"/><div style="display:block">Please click <a href="/httpservice/retry/enablejs?sei=Z3KIZ57GL8_O0PEPgcvWoQc" rel="nofollow noopener noreferrer">here</a> if you are not redirected within a few seconds.</div></noscript><div id="java-com-google-gws-common-impl-searchguard-jslayout__broken-text" style="display:none">If you're having trouble accessing Google Search, please<a href="search?q=gAAAAABniHJnrPn5yG4WhDYIxL5Nux6TBg-m4AlEgtELaocG8KstgxbYbOR-2WxjEyFneQzU7JJgOLi43_XxZmYgEDmm65Wjvw==&amp;country=&amp;tbs=" rel="nofollow noopener noreferrer">click here</a>, or send<a href="support.google.com" rel="nofollow noopener noreferrer">feedback</a>.</div></body></html>

To Reproduce
Steps to reproduce the behavior:

  1. Search for anything

Deployment Method

  • Heroku (one-click deploy)
  • Docker
  • run executable
  • pip/pipx
  • Other: [describe setup]

Version of Whoogle Search

  • Latest build from [source] (i.e. GitHub, Docker Hub, pip, etc)
  • Version [v0.9.1]
  • Not sure

Desktop (please complete the following information):

  • OS: macOS
  • Browser: Safari (any)
  • Version n/a
@buzzert buzzert added the bug Something isn't working label Jan 16, 2025
@vkoaala
Copy link

vkoaala commented Jan 16, 2025

I'm having the same issues, but I'm being rate limited as per my docker logs : ERROR:app:503 (CAPTCHA)

@buzzert
Copy link
Author

buzzert commented Jan 16, 2025

Not seeing any rate limitations mentioned in my logs.

whoogle-search    | Running on http://0.0.0.0:5000
whoogle-search    | Jan 16 02:38:17.229 [notice] Tor 0.4.8.13 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.3.2, Zlib 1.3.1, Liblzma 5.6.2, Libzstd 1.5.6 and Unknown N/A as libc.
whoogle-search    | Jan 16 02:38:17.229 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
whoogle-search    | Jan 16 02:38:17.230 [notice] Read configuration file "/etc/tor/torrc".
whoogle-search    | Jan 16 02:38:17.235 [notice] Opening Socks listener on 127.0.0.1:9050
whoogle-search    | Jan 16 02:38:17.235 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9050
whoogle-search    | Jan 16 02:38:17.235 [notice] Opening Control listener on 127.0.0.1:9051
whoogle-search    | Jan 16 02:38:17.235 [notice] Opened Control listener connection (ready) on 127.0.0.1:9051
whoogle-search    | Jan 16 02:38:17.235 [warn] Fixing permissions on directory /var/lib/tor
whoogle-search    | /whoogle/app/utils/search.py:106: SyntaxWarning: invalid escape sequence '\s'
whoogle-search    |   if match := re.search("(^|\s)!($|\s)", q):
whoogle-search    | * Finished creating ddg bangs json

@LupinThidr
Copy link

LupinThidr commented Jan 16, 2025

Not directly related to this project, but I've been getting these same redirects for the past couple weeks (on my browser with JS disabled). I've been able to "bypass" it by just simply re-querying, but as of today, they've become more persistent. After enabling Javascript, I'm redirected to the usual "google.com/sorry/index" captcha page that used to regularly happen to me on VPNs.

Relevant HN discussion from today ("Google.com search now refusing to search for FF esr 128 without JavaScript"): https://news.ycombinator.com/item?id=42719865

@mon5termatt
Copy link

mon5termatt commented Jan 16, 2025

Getting this as well, temporarily redirecting my engine to searxng but would love to get this resolved as it's too cluttered :D

was putting off making an issue while i verified it on a new install....

@scottmando2000
Copy link

Also experiencing this behavior, started happening a few days ago and has entirely stopped working today.

@philebosxxx
Copy link

I am too having exactly the same issue

@bitbucket90
Copy link

+1

@blackletum
Copy link

I am also having the same issue, on latest.

@skaboy71
Copy link

skaboy71 commented Jan 16, 2025

Same here -was able to refresh but now no-worky. Running in docker on Unraid

@damien-crypto
Copy link

Same on Umbrel OS but only in the "ALL" option.
The rest work, news, images etc.

@jbhorner
Copy link

+1

News, Images, etc. works for me as well. I tried to use by disabling tor through environment variable, but that didn't change anything.

@gads83
Copy link

gads83 commented Jan 16, 2025

+1 here as well

@abhandal
Copy link

+1 same issue

@patienttruth
Copy link

Same here since this morning. Relieved for a moment that it's not on my end, but that's actually more concerning...

@mitanshu7
Copy link

+1

2 similar comments
@conundrum741
Copy link

+1

@jersam
Copy link

jersam commented Jan 16, 2025

+1

@Nick4154
Copy link

+1 for me, Docker or Debian

@Pootch
Copy link

Pootch commented Jan 16, 2025

Same issue here. No change from my side at the docker compose.

Running on http://0.0.0.0:5000
Jan 16 07:31:43.157 [notice] Tor 0.4.8.13 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.3.2, Zlib 1.3.1, Liblzma 5.6.2, Libzstd 1.5.6 and Unknown N/A as libc.
Jan 16 07:31:43.157 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
Jan 16 07:31:43.157 [notice] Read configuration file "/etc/tor/torrc".
Jan 16 07:31:43.161 [notice] Opening Socks listener on 127.0.0.1:9050
Jan 16 07:31:43.161 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9050
Jan 16 07:31:43.161 [notice] Opening Control listener on 127.0.0.1:9051
Jan 16 07:31:43.161 [notice] Opened Control listener connection (ready) on 127.0.0.1:9051
Jan 16 07:31:43.161 [warn] Fixing permissions on directory /var/lib/tor
/whoogle/app/utils/search.py:106: SyntaxWarning: invalid escape sequence '\s'
if match := re.search("(^|\s)!($|\s)", q):

Getting this page instead:
Screenshot 2025-01-16 084711

@mooglestiltzkin
Copy link

+1 truenas docker container.

only recently occured, empty results.

@qcdesign
Copy link

+1 for me, Docker. Same as everyone. I get the 503 captcha in the docker log but blank results, Images/News etc. work fine.

@stdestro
Copy link

same here on docker. search results is blank, images/videos/news/maps works

@mztiq
Copy link

mztiq commented Jan 16, 2025

+1

2 similar comments
@Tubeman-cell
Copy link

+1

@dsvdglenn
Copy link

+1

@bruvv
Copy link
Contributor

bruvv commented Jan 16, 2025

The +1 comments are pretty useless, please there is a
image
button, use that instead.

@foegra
Copy link

foegra commented Jan 16, 2025

maybe would be good to add backup search engine?

@mztiq
Copy link

mztiq commented Jan 16, 2025

@bruvv the +1 is not about subscribing it's more for showcasing how many people are affected by this issue.

@bruvv
Copy link
Contributor

bruvv commented Jan 16, 2025

@mztiq please use emoticons then:
image
This is just spamming @benbusby i think he now knows about this issue.

@xircon
Copy link

xircon commented Jan 17, 2025

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Vivaldi/7.0.3495.29"

@benbusby - Vivaldi UA works - I use Vivaldi.

@talentedmrripley
Copy link

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Vivaldi/7.0.3495.29"

@benbusby - Vivaldi UA works - I use Vivaldi.

This one isn't working for me. I've tested several of the others and they all return results but this one doesn't.

@xircon
Copy link

xircon commented Jan 17, 2025

Maybe because I am using vivaldi???

    environment: # Uncomment to configure environment variables
      #- WHOOGLE_USER_AGENT="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0" 
      - WHOOGLE_USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Vivaldi/7.0.3495.29"
      #- WHOOGLE_USER_AGENT_MOBILE="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0" 

      #- WHOOGLE_USER_AGENT="Mozilla/3.0 (compatible; MSIE 3.0; Windows NT 5.0)"
      #- WHOOGLE_USER_AGENT_MOBILE="Mozilla/3.0 (compatible; MSIE 3.0; Windows NT 5.0)"

      - WHOOGLE_USE_CLIENT_USER_AGENT=0

The above is my docker-compose.yml snipped.

@jeffbl
Copy link

jeffbl commented Jan 17, 2025

@calderjh
Copy link

calderjh commented Jan 18, 2025

Any pointers on how to implement this fix? I'm afraid I'm clueless! I installed in proxmox as an LXC and that was about the extent of my technical input!

@cauliflowerdoughnuts
Copy link

Any pointers on how to implement this fix? I'm afraid I'm clueless! I installed in proxmox as an LXC and that was about the extent of my technical input!

I'm assuming you used the Proxmox VE Helper-Scripts? Just re-run their install command and it will pull the update via pip.

@VenomousRage
Copy link

Any pointers on how to implement this fix? I'm afraid I'm clueless! I installed in proxmox as an LXC and that was about the extent of my technical input!

I'm assuming you used the Proxmox VE Helper-Scripts? Just re-run their install command and it will pull the update via pip.

I am in a similar situation. I just tried the in-built "update" command, as well as then deleting the LXC and spinning up a fresh new one with the install script, and neither worked. The issue remains the same. The "all" tab is empty. The other tabs all populate results.

Perhaps when the community script is ran, it pulls a cached copy of Whoogle to install and not a fresh one?

@calderjh
Copy link

Any pointers on how to implement this fix? I'm afraid I'm clueless! I installed in proxmox as an LXC and that was about the extent of my technical input!

I'm assuming you used the Proxmox VE Helper-Scripts? Just re-run their install command and it will pull the update via pip.

No didnt work for me - also tried "update" - nothing - same results as VenomousRage

@calderjh
Copy link

I did try to search for the modified file so I could just overwrite it but there are multiple versions and I didnt know which to replace - this is via "filebrowser" - also installed as a proxmox script into the LXC whoogle lives in.

@cauliflowerdoughnuts
Copy link

Any pointers on how to implement this fix? I'm afraid I'm clueless! I installed in proxmox as an LXC and that was about the extent of my technical input!

I'm assuming you used the Proxmox VE Helper-Scripts? Just re-run their install command and it will pull the update via pip.

I am in a similar situation. I just tried the in-built "update" command, as well as then deleting the LXC and spinning up a fresh new one with the install script, and neither worked. The issue remains the same. The "all" tab is empty. The other tabs all populate results.

Perhaps when the community script is ran, it pulls a cached copy of Whoogle to install and not a fresh one?

@calderjh if the update worked you should have a user agent drop down option in the config on the whoogle landing page and it should default to lynx which works. if you dont have that drop down then the update didnt work.

@calderjh
Copy link

calderjh commented Jan 18, 2025

@calderjh if the update worked you should have a user agent drop down option in the config on the whoogle landing page and it should default to lynx which works. if you dont have that drop down then the update didnt work.

OK that's not there and version still showing as v0.9.1 (current) - from what I can see, just one file has been updated, do you know where this is located exaclty? I could just drag-drop the new one over?

@cauliflowerdoughnuts
Copy link

@calderjh if the update worked you should have a user agent drop down option in the config on the whoogle landing page and it should default to lynx which works. if you dont have that drop down then the update didnt work.

OK that's not there and version still showing as v0.9.1 (current) - from what I can see, just one file has been updated, do you know where this is located exaclty? I could just drag-drop the new one over?

try running pip3 install whoogle-search --upgrade in your LXC container

@calderjh
Copy link

pip3 install whoogle-search --upgrade

Gives me this:

Last login: Sat Jan 18 00:58:20 GMT 2025 on tty1
root@whoogle:# pip3 install whoogle-search --upgrade
Requirement already satisfied: whoogle-search in /usr/local/lib/python3.11/dist-packages (0.9.1)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (4.12.3)
Requirement already satisfied: brotli in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (1.1.0)
Requirement already satisfied: cssutils in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (2.11.1)
Requirement already satisfied: cryptography in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (44.0.0)
Requirement already satisfied: defusedxml in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (0.7.1)
Requirement already satisfied: Flask in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (3.1.0)
Requirement already satisfied: python-dotenv in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (1.0.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from whoogle-search) (2.28.1)
Requirement already satisfied: stem in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (1.8.2)
Requirement already satisfied: validators in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (0.34.0)
Requirement already satisfied: waitress in /usr/local/lib/python3.11/dist-packages (from whoogle-search) (3.0.2)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.11/dist-packages (from beautifulsoup4->whoogle-search) (2.6)
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.11/dist-packages (from cryptography->whoogle-search) (1.17.1)
Requirement already satisfied: more-itertools in /usr/local/lib/python3.11/dist-packages (from cssutils->whoogle-search) (10.5.0)
Requirement already satisfied: Werkzeug>=3.1 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (3.1.3)
Requirement already satisfied: Jinja2>=3.1.2 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (3.1.4)
Requirement already satisfied: itsdangerous>=2.2 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (2.2.0)
Requirement already satisfied: click>=8.1.3 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (8.1.7)
Requirement already satisfied: blinker>=1.9 in /usr/local/lib/python3.11/dist-packages (from Flask->whoogle-search) (1.9.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.11/dist-packages (from cffi>=1.12->cryptography->whoogle-search) (2.22)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/dist-packages (from Jinja2>=3.1.2->Flask->whoogle-search) (3.0.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@whoogle:
#

But still not working

@dave0003
Copy link

For what it's worth, "pip install whoogle-search --upgrade" also doesn't seem to be pulling any updates for me, still at 0.9.1.

I also usually get the latest requirements and update them.

wget https://raw.githubusercontent.com/benbusby/whoogle-search/main/requirements.txt
pip install -r requirements.txt --upgrade

It did get some new cssutils.

But I did get it working by manually adding the user agent to the environment:

whoogle/lib/python3.11/site-packages

WHOOGLE_USE_CLIENT_USER_AGENT=0
WHOOGLE_USER_AGENT="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"
WHOOGLE_USER_AGENT_MOBILE="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"

@calderjh
Copy link

calderjh commented Jan 18, 2025

For what it's worth, "pip install whoogle-search --upgrade" also doesn't seem to be pulling any updates for me, still at 0.9.1.

I also usually get the latest requirements and update them.

wget https://raw.githubusercontent.com/benbusby/whoogle-search/main/requirements.txt
pip install -r requirements.txt --upgrade

It did get some new cssutils.

But I did get it working by manually adding the user agent to the environment:

whoogle/lib/python3.11/site-packages

WHOOGLE_USE_CLIENT_USER_AGENT=0
WHOOGLE_USER_AGENT="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"
WHOOGLE_USER_AGENT_MOBILE="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"

Thanks so much but I can't find site-packages for the life of me. In that location I have sitecusomize.py but not sure i should be meddling with this if its the wrong one!

@bullfrog-au
Copy link

New install using docker-compose and the user agent env variables, I get an error on creating the container.
[+] Running 0/1
⠋ Container whoogle-search Creating 0.0s
Error response from daemon: invalid mount path: 'WHOOGLE_USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Vivaldi/7.0.3495.29"' mount path must be absolute

Docker compose yaml
- WHOOGLE_USE_CLIENT_USER_AGENT=0
#- WHOOGLE_USER_AGENT="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"
- WHOOGLE_USER_AGENT_MOBILE="Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"
- WHOOGLE_USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Vivaldi/7.0.3495.29"

@dave0003
Copy link

site-packages is a directory. I should have added the filename.

whoogle/lib/python3.11/site-packages/whoogle.env

You might need to add the file. But your setup might be different, too. I'm still not sure why it's not pulling any update.

@calderjh
Copy link

calderjh commented Jan 18, 2025

Totally different set-up I think as nothing resembling this exists. The version number has not been incremented so i would assume that the old version is still being dished out? But I am ignorant of how GitHub works so could be wrong.

@bullfrog-au
Copy link

Totally different set-up I think as nothing resembling this exists. The version number has not been incremented so i would assume that the old version is still being dished out? But I am ignorant of how GitHub works so could be wrong.

Looking at the current main branch on github the active version is '0.9.1'

Image

@calderjh
Copy link

0.91 was released on 24th October last year. Which, I guess, is why we have no fix in that version - I just wasn't sure if the code was a "living entity" as such whereby any requests to download automatically fed from an updated version.

Seems not.

@golesscom
Copy link

Programmatically decode and decipher the code provided by Google to determine if JavaScript is enabled, and then save this code in cookies or local storage as requested.

@xaabi6
Copy link
Contributor

xaabi6 commented Jan 18, 2025

For all of those using Whoogle LXC container by Proxmox VE Helper-Scripts, I've updated requirements as @dave0003 suggests:

For what it's worth, "pip install whoogle-search --upgrade" also doesn't seem to be pulling any updates for me, still at 0.9.1.

I also usually get the latest requirements and update them.

wget https://raw.githubusercontent.com/benbusby/whoogle-search/main/requirements.txt
pip install -r requirements.txt --upgrade

It did get some new cssutils.

Then, adding environment variables was quite different. I achieved it by editing the service using systemctl edit whoogle.service command:

[Service]
Environment="WHOOGLE_USE_CLIENT_USER_AGENT=0"
Environment="WHOOGLE_USER_AGENT=Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"
Environment="WHOOGLE_USER_AGENT_MOBILE=Lynx/2.9.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/3.4.0"

Then restart the service issuing systemctl restart whoogle.service and you're done. Hope it helps someone!

@dave0003
Copy link

That makes sense. In my whoogle.service file I've got:

Environment=WHOOGLE_DOTENV=1

I think that tells it to use the whoogle.env file, but it looks like you can add them all there, too. I use it to set some other settings, too, so it's handy.

@xaabi6
Copy link
Contributor

xaabi6 commented Jan 18, 2025

You're right, it can also be done with a dedicated file for environment variables, but as the service needs to be edited for both types of configuration, I prefered to put all of them in the same place 😉

@bitchecker
Copy link

I've been experiencing the same issue!

I've also configured WHOOGLE_FALLBACK_ENGINE_URL=https://duckduckgo.com/?k1=-1&q= to try to work around, but it doesn't seem to work (maybe my config is wrong, but it what is reported on readme).

Maybe a configuration to change the backend search engine could be helpful.

@patienttruth
Copy link

Hi just tried your fix and working great. Thank you.

From the command prompt in the environment you're using to host Docker:

(Run these in the directory where your Docker Compose file is located.) docker compose down docker compose pull docker compose up -d

This worked great though now I'm seeing mention of TOR in the logs. I don't believe that was there before.

Running on http://0.0.0.0:5000
Jan 19 04:03:36.567 [notice] Tor 0.4.8.13 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.3.2, Zlib 1.3.1, Liblzma 5.6.2, Libzstd 1.5.6 and Unknown N/A as libc.
Jan 19 04:03:36.568 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
Jan 19 04:03:36.569 [notice] Read configuration file "/etc/tor/torrc".
Jan 19 04:03:36.583 [notice] Opening Socks listener on 127.0.0.1:9050
Jan 19 04:03:36.583 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9050
Jan 19 04:03:36.583 [notice] Opening Control listener on 127.0.0.1:9051
Jan 19 04:03:36.583 [notice] Opened Control listener connection (ready) on 127.0.0.1:9051
/whoogle/app/utils/search.py:106: SyntaxWarning: invalid escape sequence '\s'
  if match := re.search("(^|\s)!($|\s)", q):
Setting custom user agent to: 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working unfortunate Something is very unfortunate
Projects
None yet
Development

No branches or pull requests