Missing query params in url when params option is set

[yanyongyu]

In version 0.28.0, query params are never merged.

import httpx

client = httpx.Client()
request = client.build_request(
    "GET", "https://httpbin.org/get?param=default", params={"foo": "bar"}
)
print(request.url)  # This will be `https://httpbin.org/get?foo=bar`

The query string in the request url will be ignored. It seems this is caused by the pr #3364. Params are replaced instead of merged when parsing.

httpx/httpx/_urlparse.py

Line 301 in 15e21e9

query = kwargs.get("query", url_dict["query"])

Related discussion #3428

[tomchristie]

Params are replaced instead of merged when parsing.

This is exactly the behavior I would expect.
As I see it, the simplification in #3364 introduced a bugfix that we haven't documented in the release.

[yanyongyu]

I know providing both params and query in url is a bad practice. But this can be counterintuitive in some situations like:

import httpx

client = httpx.Client(params={"foo": "bar"})
request = client.build_request(
    "GET", "https://httpbin.org/get?param=default"
)
print(request.url)  # This will be `https://httpbin.org/get?foo=bar`

The params are provided in the client options. This will also replace the query in the url.

May be a warning can be raised when the replaced query is not empty?

httpx/httpx/_urlparse.py

Line 301 in 15e21e9

query = kwargs.get("query", url_dict["query"])

[tomchristie]

Related issue... Client(params=...) is a not-good idea.

But this can be counterintuitive in some situations like

That is surprising behaviour, yes. I'd consider that a bug.

To the extent that we do currently support Client(params=...) we should probably? be merging those with any params=... passed by the user. Tho it's a bad pattern in the first place that we ideally could have avoided.

[yanyongyu]

Currently, as a workaround, we merge all the params by ourselves before passing to httpx. The code is clearer logically 😂 .

[mcgfeller]

Broke quite some of my code, so at least a warning and a hint on how best to merge parameters should be issued.

[eliasdorneles]

Beware, this breaks compatibility with Requests, which does merge the parameters:

[zanieb]

It seems like it may be worth restoring the previous behavior for parity with requests? Is there a way we can discourage the "bad pattern" while still merging?

[jhominal]

Personally, I believe that the "merge" behaviour is more intuitive for someone who specifies both a url with query parameters and a params dictionary.

I often encounter the case of a GET web service, that can be implemented by many different actors, who will give me a URL string that I will stick in a configuration.
And then, in order to call that web service, I will need to add some parameters that are specific to the call (a user ID, search parameters, etc.)
With the replace behaviour being the default, this means that, in order to handle the case where the original URL contains a query string, I need to do one of these things:

1. parse the URL and put everything in params so that I do not lose anything (taking in account that httpx query params handling is not necessarily round-tripping for every query string, which can sometimes be important);
2. parse the URL and use the copy_with_merge_params manually;
3. detect whether the URL contains a ? and add a string suffix to the URL based on the parameters I want to add;

No matter the solution we recommend for merging, the params argument seems like a trap: params works only if I can be sure that my URL never has a query string - otherwise I need to do the merging dance systematically.

It would be a more honest API, in that case, if the params argument was removed and we said "you need to add the parameters to the URL in order for them to work". However, we should not do that because that would be a significant usability regression.

[urschrei]

The new behaviour also makes it more difficult to work with link URLs (e.g. prev, next, last) in http responses – I'm surprised it hasn't been brought up before. Now, extra logic is required to convert the query parameters in each link to a dict and then merge that with any other query parameters that might be required. We aren't short of tools to do that in the stdlib, but adding new logic like this requires extensive testing to ensure correctness because there are several corner cases.