reduce concurrent access throttling to 5.

[yuyiguo]

No description provided.

[amaltaro]

This is something I'm afraid of.
Throttling at 5 concurrent requests per backend will likely be an issue for MicroServices, where the current - non configurable - number of DBS concurrent requests is set to 50.

Yuyi, have we had any recent issues of clients hammering DBS? If not, then can we keep the current throttling as is?

[muhammadimranfarooqi]

@yuyiguo

This change should also be done for k8s configuration here.

https://gitlab.cern.ch/cmsweb-k8s/services_config/-/blob/preprod/dbs-global-r/DBSGlobalReader.py

[yuyiguo]

@amaltaro

I am having problem with high number of accesses from a client who is an individual. I think I sent you, Stefano and Valentin email that I'd like to reduce the number of concurrent accesses to DBS a few week ago, but I did not hear anything back from you . So I thought that it might not a problem for you.

What DN MicroServices is using? Let me check it.
The max 50 is configured, but how many does it really use?

@vkuznet @belforte
What is your max concurrent access to DBS?

Thanks,
Yuyi

[yuyiguo]

@muhammadimranfarooqi
Let's do VM first. If it works, we move to k8s.

[amaltaro]

We deal with workflows in slices of 100 workflows, so we can easily reach the limit of 50 concurrent calls. Limiting central production - which will use the same DN (CMSWEB service DN) as used by a few other cmsweb services - is too low and a very aggressive limit IMO.

I guess you already tried contacting the user?
Sorry for not getting back to you before.

[vkuznet]

Yuyi, we discussed concurrency a long time ago and we agreed that DBS can sustain a load from 100 concurrent clients. DAS uses this number fo a long time. We even made this as default in WMCore pycurl module, see https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/pycurl_manager.py#L371

Therefore I don't understand why you're complaining about limit of 50? I also don't know how you measured this and how you found that it is a problem. I also don't know how MS uses this, i.e. do multiple threads (services) allowed to have the same limit concurrently, etc.

If I recall correctly we have 4 or 5 backends for DBS, which means that you're asking to limit DBS to 25 or less concurrent requests.

I suggest that you provide full details in this thread to understand who is a client and how you measure the impact, which APIs are affected, etc. Without a full picture, I doubt that making any changes will have a meaningful impact on the overall performance of the DBS and all clients.

[belforte]

anyhow, CRAB current;y runs with 6 submission threads, so 6 queries at same time is maximum. add tests, preprod instances.. whatever... there should never be a problem, we do not fork parallel queries, each CRAB worker makes one query after another. If DBS has problems, it can't be us.

I agree with Alana that individual users should be throttled down before production, since they are the ones who usually do not know what they are doing.

[yuyiguo]

@amaltaro

We are not trying to limit the production system. The limit is based on DN and it does not know if this is a production system or not. That is why I send email to you guys before this test. Two questions: 1. How difficult for the production uses different DNs for different subsystem? This will make our monitoring giving more accurate info. 2. Is it possible you can fire tests against cmsweb testbed to see how it does with throttling is 5 (total 25=5x5instances)?

Also the limit is only for prod reader. The writer is not changed. I understood that production system has a lot of writer. What is the precent of readers for production? We have been running with limit 10 (total 50=10x5 instances) concurrent reader per DN since last release and there is no one complain that their access was blocked so far. However, we got some one with 200k/hr accesses to DBS with limit 10.

We have time to get this right for this release. You don't need to worry about production get cut.

[yuyiguo]

@vkuznet

I don't know where you got " DBS can sustain a load from 100 concurrent clients".

When DBS allowed 75 concurrent accesses, it could not sustain the load from unified.

[yuyiguo]

@belforte

Thanks for reporting on crab. We have one less system to worry about.

I agree with you and Alan that individual users should be throttled down before production, but our system is not build to identify DN.

[belforte]

cmsweb frontend maps different DN's to different users for the backend.

…

On 15/06/2020 21:01, Yuyi Guo wrote: our system is not build to identify DN

[vkuznet]

Yuyi, I don't know if I can find full email thread, but at least I found your email from 2017 which you sent to me/Eric/Alan/Seangchan (I took it from my Gmail archive):

Yuyi Guo <[email protected]> | Wed, Aug 30, 2017 at 6:28 PM
To: "Valentin E. Kuznetsov" <[email protected]>Cc: Eric W Vaandering <[email protected]>, Alan Rodrigues <[email protected]>, Seangchan Ryu <[email protected]>

Valentin:

I remember that we set down the concurrent connect was 100 for DAS back a few years ago. I also recalled that you had an high number that DBS could not support that. If we implement this concurrent API in a general way that means it would be more people will use it so we need to make sure DBS can handle it.

I am also interested to compare the two different connections. We could make a test plan together.
Cheers,
Yuyi

As you can see we had discussion about concurrency and 100 number was there. That's why we set it up in pycurl module at that time. As I wrote the 100 limit was discussed and you agreed to have it at that time. Most likely the overall load on DBS has increased since then and new limit is required, but I didn't invent (even though I did propose it) and it was set based on our discussions and your agreement.

[amaltaro]

Let me try to answer/comment on a few points here.

I also don't know how MS uses this, i.e. do multiple threads (services) allowed to have the same limit concurrently, etc.

For now, it's only MSTransferor process using the pycurl module to create up to 50 concurrent requests. As the system evolves and become more scalable, we will adopt pycurl concurrent calls in different services as well (like global workqueue, future microservices to be implemented, etc).

We are not trying to limit the production system. The limit is based on DN and it does not know if this is a production system or not.

I know, but since it's a system-wide throttling, central production is implicitly throttled too.

1. How difficult for the production uses different DNs for different subsystem?

Very much! CMSWEB services - requiring a grid certificate to authenticate to other cmsweb services - all use the same "dmwm/cmsweb.cern.ch" certificate (or the one correspondent to the k8s cluster). Having different DNs will increase the load on the CMSWEB team, which will have to manage multiple certificates.
It might not help monitoring, but I do think we should set different User-Agent for different WMCore services (not sure how straight forward that change can be done though...)

2. Is it possible you can fire tests against cmsweb testbed to see how it does with throttling is 5 (total 25=5x5instances)?

That's of course possible, but it would be very labor intensive to come up with a load similar to what we can have in production. In testbed, we do not run more than a couple dozen workflows at a given time.

BTW, testbed has only two instances, so the maximum number of concurrent calls would actually be 10.

Also the limit is only for prod reader. The writer is not changed. I understood that production system has a lot of writer.

It looks like (each) agents are configured to run 4 processes for DBS data injection. We usually have 2-5 agents under the same DN.
The Reader endpoint will be used for data discovery, and later this year for workflow completion as well, where I foresee a high number of concurrent calls to be made.

We have been running with limit 10 (total 50=10x5 instances) concurrent reader

Just to be clear, 50 is the very maximum limit here; throttling could get triggered as soon as we have 15 or 20 concurrent requests in the system (depending on the frontend load balancing; number of backends running at any given time)

However, we got some one with 200k/hr accesses to DBS with limit 10.

I assume the API(s) he/she was accessing were pretty quick. Assuming each request took a second, he/she would have to run around 55 concurrent requests (which should have triggered even the max limit of 50 calls).

[vkuznet]

Despite the desire to understand the issue, I see that proposed changes do not address the main issue discussed here. You may like it or not, but the load has increased, and will increase in the future. Reducing the throttling limit does not resolve the underlying issue with DBS performance and I rather prefer to hear discussion of how to improve it to accommodate the increasing needs of the experiment and our data management system.

Even though the throttling should be in place it should not substitute the effort to increase DBS concurrency throughput.

From what I know the python should be capable to sustain 10-20 concurrent calls per backend. This brings us to 50-100 concurrency limit. Why we're discussing a limit of 5 is not clear to me, neither I see clear details which DBS APIs can't sustain that load. It may likely be the case that instead of throttling we need DBS API change or switching from CherryPy to Flask+wsgi or more drastically change. But this ticket does not provide enough details about the original issue.

[belforte]

On 16/06/2020 02:47, Alan Malta Rodrigues wrote: For now, it's only MSTransferor process using the pycurl module to create up to 50 concurrent requests. As the system evolves and become more scalable, we will adopt pycurl concurrent calls in different services as well (like global workqueue, future microservices to be implemented, etc).

the Unified way ? Since one call is not fast enough for my taste, let's make hundreds at a time so the server dies on DOS ? I do not understand why so much DBS traffic is needed to PRODUCE datasets, Is this writing, or reading ? is WMA using DBS for internal bookkeeping ? Or to monitor transfers ? I am not w/o sin here, CRAB use of its internal DB for bookkeeping is way from sensible, not to say optimized (not my code, but this does not help) so I shouldn't be throwing stones, but realistically I think that more care in WMA is more likely to be possible and help than redesigning DBS API's. Also it should be quite easy to use different credentials (cert.) to talk with DBS and to talk with WMA MS's. CRAB already uses different DN's for CMSWEB, myproxy and HTCondor, so I think we understand by now the way to do this and surely can share. Bottom line IMHO: short of a _major_ intervention from DMWM L2's and L1's and serious injection of manpower, we should better assume that DBS is what it is and try to live with it. The only action that I would do centrally it so separate production and user traffic on different servers, let users suffer, and protect production.

[amaltaro]

the Unified way ? Since one call is not fast enough for my taste, let's make
hundreds at a time so the server dies on DOS ?

nope, we do not make uncontrolled number of concurrent requests to DBS, as it happened to Unified in the past.

I do not understand why so much DBS traffic is needed to PRODUCE datasets,
Is this writing, or reading ?

Because we need to run data discovery. How do you find out which files exactly have to be processed if you get a workflow configured like:
dataset: /Muon/blah/RAW
LumiList: {"123": [[222, 333], [400, 510]], "234": [[1233, 1244]]}

or if you have a run white list? That takes quite some DBS queries!
In addition to that, as I said before, we run external service calls in slices of 100. So if I need to check what blocks are present in 100 datasets, there you go with 100 DBS calls (well, 50, because this is the current limit).

is WMA using DBS for internal bookkeeping ?

WMAgent is a stateful machine, so we do need to keep record of input data information (blocks, files, runs, lumis, num_events), such that we can properly inject output data into DBS.

Or to monitor transfers ?

Of course not! You know this is not possible ;)

Also it should be quite easy to use different credentials (cert.) to talk with
DBS and to talk with WMA MS's. CRAB already uses different DN's for CMSWEB,
myproxy and HTCondor, so I think we understand by now the way to do this
and surely can share.

WMCore MS live in the cmsweb ecosystem (they are deployed in cmsweb nodes).

[vkuznet]

Stefano, the concurrent architecture of our services is now a necessity. We can't live anymore in sequential design since it is too slow. I don't know all details of production/unified/WMCore/MicroServices but I do know that in order to run workflows or produce data we need a data look-up, e.g. look-up of datasets/blocks/files/runs/lumis. These queries can be run sequentially but then underlying production machinery is slow, or we can run things concurrently and we can significantly speed-up production. Therefore, I would not say that leaving with DBS "as is" is a way we should operate. My point is that we should strive to improve our systems not because we may do that but because we need/must to do it now in order to cope up with the increasing load.

[belforte]

On 16/06/2020 09:30, Alan Malta Rodrigues wrote: WMCore MS live in the cmsweb ecosystem (they are deployed in cmsweb nodes).

sure, but you can certainly imagine to switch credentials when you talk to DBS and when you talk to somebody else. I wrapped CRAB calls with a context handler, but maybe you can do even better...

[belforte]

Oh man.. I would have hoped this to be some very exceptional and rare circumstance, I mean.. rare as "100 times a day", not 100 times a minute ! But if that's the world you need to live in, surely it is a difficult place to be !

…

On 16/06/2020 09:30, Alan Malta Rodrigues wrote: How do you find out which files exactly have to be processed if you get a workflow configured like: dataset: /Muon/blah/RAW LumiList: {"123": [[222, 333], [400, 510]], "234": [[1233, 1244]]}

[belforte]

I am not disagreeing with that. Simply worrying at the amount of needed work. But it is quite likely that my personal view of what can be done is limited by my limited skills !!!

…

On 16/06/2020 09:36, Valentin Kuznetsov wrote: Therefore, I would not say that leaving with DBS "as is" is a way we should operate. My point is that we should strive to improve our systems not because we may do that but because we need/must to do it now in order to cope up with the increasing load.

[yuyiguo]

Thanks everyone for the discussion.
@vkuznet , The situation changed a lot for the past two years, we cannot base on the old data any more. As you may see that in the stepChain, we were able to find out the parentage in one API call in the past, but we now have to separate into two APIs and the matching has to do in python instead of DB. If one wants to increase DBS's responding rate, one has to work on the database side, not rewrite DBS in a different language. I am up to improve DBS. I talked with our L2 leaders about this.

@belforte Thanks for providing info on crab using different DNs. This is really good to know. Your experience will be useful to others.

@amaltaro
In addition to that, as I said before, we run external service calls in slices of 100. So if I need to check what blocks are present in 100 datasets, there you go with 100 DBS calls (well, 50, because this is the current limit).

Alan, How often the external service makes 100 concurrent DBS calls? This is additional to the 50 concurrent calls of MS? All these were using the same DN? And you did not get throttled! That meant the throttling code did not do what it supposed to do?

I looked into the monitoring, there are different DNs used for production, such as CN=wmagent/vocms0280.cern.ch and CN=dmwm/cmsweb.cern.ch . As Stefano pointed out that crab already uses different DNs and it did not seem to me that was very hard to achieve. When we talked about where the DBS calls of the production systems from, it did not appear to me that we really knew them very well. That is where the monitoring can help, but we have to make some adjustment to make that happen. I am not saying that you need to do it now, but you may want to do it when you are building new system or updating the existing service, such as MicroServices.

I knew you got a lot of stuff in your plate. But I still ask if you could put some tests against DBS testbed. Since there are only 2 instance, could you run the external service that calls 100 concurrent APIs ?

[amaltaro]

@yuyiguo I was pretty sure the throttling in pycurl had been lower to 50, but as I look at the code now:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/pycurl_manager.py#L371

I still see a default of 100 concurrent calls. I'm going to lower that default to 50 only.

And yes, the MS deployed in CMSWEB uses the same DN all the time. WMAgent, on the other hand, use a different DN (because they are not deployed in CMSWEB, and because they are made available at FNAL and CERN)

I knew you got a lot of stuff in your plate. But I still ask if you could put some tests against DBS testbed. Since there are only 2 instance, could you run the external service that calls 100 concurrent APIs ?

I can test it, but I doubt I could run 100 concurrent DBS calls. It would require me to clone a bunch of workflows from the production system and get them running in testbed. But I'll let you know what I can reach.

[yuyiguo]

@amaltaro
The throttling in pycurl has been set to 100 as default, but the system never research the default value, right? The highest we had in DBS was 75 for past two years, then it was set to 50 since the June release.

Thanks for testing !

[vkuznet]

I think we prematurely close this PR. The issue did not provide any details on why we should lower the throttling limit. I suggest to postpone the inclusion of this PR into production until we will have the following information:

• @yuyiguo should provide complete details about the incident(s), this should include;
  • list of DN(s)/client(s)/User-agent(s) who introduced the problem to DBS
  • list of APIs which cause the problem
  • time frame(s) when it happens and frequencies

All this information is required to make proper decision, otherwise, I fear that such a reduction will have a negative impact on different systems/use-cases, including production services such as ReqMgr/MS/DAS/.... We should evaluate the situation and decide if it is t worth reducing the throttling limit or try to find an alternative solution, e.g. add another node to handle the load.

[yuyiguo]

Valentin, We are not going to move this to production before we understand the throttling and access rate. In order to do so, we have to deploy this on testbed for the study. Muhammad is applying this to the testbed only. From: Valentin Kuznetsov <[email protected]> Reply-To: dmwm/deployment <[email protected]> Date: Wednesday, June 17, 2020 at 7:46 AM To: dmwm/deployment <[email protected]> Cc: Yuyi Guo <[email protected]>, Mention <[email protected]> Subject: Re: [dmwm/deployment] reduce concurrent access throttling to 5. (#919) I think we prematurely close this PR. The issue did not provide any details on why we should lower the throttling limit. I suggest to postpone the inclusion of this PR into production until we will have the following information: · @yuyiguo<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_yuyiguo&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=KnTGAvyLhQVmSUG4Uj_BuTaiPb2KoG-soFzSmazDkfY&s=8yNQ9Ukhj7iOd_AxJEvnZRtulY-fYibAueJLbSo4aT8&e=> should provide complete details about the incident(s), this should include; o list of DN(s)/client(s)/User-agent(s) who introduced the problem to DBS o list of APIs which cause the problem o time frame(s) when it happens and frequencies All this information is required to make proper decision, otherwise, I fear that such a reduction will have a negative impact on different systems/use-cases, including production services such as ReqMgr/MS/DAS/.... We should evaluate the situation and decide if it is t worth reducing the throttling limit or try to find an alternative solution, e.g. add another node to handle the load. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_deployment_pull_919-23issuecomment-2D645351788&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=KnTGAvyLhQVmSUG4Uj_BuTaiPb2KoG-soFzSmazDkfY&s=AGCOxDo20x93pOqi42Sxw0uW0FrK5r3PQApojCVyu5U&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AANROTVDZNNVJ2G2AQI3HBDRXC3LBANCNFSM4N6UAWKQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=KnTGAvyLhQVmSUG4Uj_BuTaiPb2KoG-soFzSmazDkfY&s=NxEcaYDEtsg7yGJBvJvQ6G-c95uDpD5_wbxuCqbRa5Q&e=>.

[muhammadimranfarooqi]

@yuyiguo

This PR has been deployed on testbed today.

[vkuznet]

Yuyi, thanks for the clarification.

Another aspect, do you consider putting time-consuming DBS APIs into dedicated service such that we can effectively scale DBS at API level? For instance, I doubt that the APIs you prepared for Micro-Service (MS) use-case will be used by the average user. Doing this way we may still have a higher throttling threshold for most common APIs while lower it for more time-consuming ones. Yet, another approach would be to start applying MS idea to DBS APIs. For instance, in k8s we can have dedicated pods for single DBS API end-point and then we can scale APIs independently. This approach does not require major code refactoring but will require a separation of DBS server into multi-services which we can then deploy independently in k8s and use k8s scaling capabilities for DBS micro-services.

[klannon]

These are good suggestions. It sounds to me like moving DBS to K8s ASAP would open up a lot of flexibility for how we address these issues. @yuyiguo: Do you think we can get to a relatively stable state and then focus efforts on the K8s migration?

[yuyiguo]

@vkuznet
I was not sure I understood you suggestion. How to dedicated pods to certain APIs? Is the cmsweb front end can control which API goes to which pod?

@klannon
Adjusting the throttling has been in the plan for a while. It just that people started to pay attention to it. We need a suitable throttling even in k8s.

[vkuznet]

Yuyi, it is not about dedicated pods, it is about dedicated DBS instances. Now you have different instances related to different DB backends or operations, e.g. DBS global vs DBS phys01, or DBSReader vs DBSWriter. So if you can create different servers that will only serve single API then those services can be deployed independently in k8s which will allow you to scale them independently. Since services will run independently (literally different python processes), then we can adjust cmsweb frontend rules to reach them on their dedicated ports.

[belforte]

@yuyi I guest that the answer to this

. How to dedicated pods to certain APIs? Is the cmsweb front end can control which API goes to which pod?
is that it can be done via redirection in the frontend as long as one API maps to a different path in the query URL. Just like the frontend decides where to map cmsweb,cern.ch/dbs and cmsweb.cern.ch/reqmgr2 it can map cmsweb,cern.ch/dbs/filesummary to one place and cmsweb,cern.ch/dbs/datasets to another

[yuyiguo]

We may try per API per instance/pod, but it we are going to end with a lot of pods/instances. It more make sense to me that certain instances/pods to certain clients, such as production system has its own instances/pods, crab has its own and other users have their own.

Anyway, all is worth a try.

[belforte]

that'd be step 1: separate traffic. Then breaking the server into microservices which can be individually scaled horizontally would be step 2. From the outside is will look the same as running many DBS servers, but should require much less hardware.