Shall we set it a bit larger, e.g. 60, as to include 99% of cases instead? Or you say that eventually one would just restart the calculation if the slope is good?

That was my thinking. And we would be a bit more pre-emptive in fixing ones that do not, so potentially save time that would otherwise be wasted. In the end I suppose the question is what we prefer: avoiding restarts or avoiding wasted resources? With a slope threshold of -0.1, it's very likely that we'll catch all the calculations who just need more steps, and those rare edge cases might still benefit from some reduced mixing and hence not be lost.

That said, 60 is probably also fine (it includes 98.8% of cases, so 1.3% more ^^).

Then again... I was redoing the analysis on the slopes, but then instead of looking at the slopes over all steps, look only at the 50 we actually calculated in case we'd set electron_maxstep to 50:

Now we see quite a few with slopes > -0.1 that do converge. Looking at the evolution of some of their scf accuracies:

We can see that indeed, only after around step 50 the SCF "catches its groove", and the convergence begins accelerating. Maybe this would also happen if we restart from the charge density and reduce the mixing, but that isn't certain and it'd most likely be slower.

You can play round with the "max nstep" to consider. Of course as you increase it, you have fewer cases where the slope before "max nstep" is > -0.1 and the calculation converges, but you have fewer cases of successful convergence between "max step" and 80 steps as well.

So we could ask two questions: if I would have set electron_maxstep to X, and used -0.1 as the slope threshold

• what's the chance that we did an "incorrect stop", i.e. the slope is > -0.1, but the calculation converges in 80 steps?
• what's the chance that we did an "incorrect restart", i.e. the slope is < -0.1, but the calculation fails to converge in 80 steps?

But frankly, these % data are also skewed by the fact that we only have data up to step 80, i.e. if we had data up to 200 steps the % incorrect stop would go down for all "max nstep", but most likely more for 70 than for 40.

So to do this analysis "right", we'd need to gather statistics for a larger electron_maxstep. Moreover, the question is what happens if we "incorrectly stop" a calculation and decide to e.g. reduce the mixing. It's all quite a lot of analysis without too much benefit in the end, I think. 60 seems like a reasonable choice to me, so I'll pick that unless there are objections.

One note though: it's clear from the "scf accuracy" curves above that extrapolation may not always give a reasonable value. I'd keep it simple: if the slope < -0.1, just double the electron_maxstep to 120 and restart from the last charge density.

Thanks @mbercx for the analysis. I must say I couldn't completely follow the arguments. But the figure you showed is also interesting. Maybe we should compute the slope only on e.g. 20 of the last steps, and look at the slope, and decide whether to restart or to change the mixing mode?

Yeah, it was a little late, so maybe my explanation wasn't the best. ^^ Let me summarize:

In my previous analysis I used the slope over all 80 steps to see if we can decide after 50 steps to continue with more steps or switch to another strategy. That's of course not correct, since at step 50 we'd only have information about the first 50. If we look at the slope distributions for the first 50 steps (first image in my post above), we can see that there are more cases where selecting -0.1 as the threshold would lead to our approach switching to another strategy (e.g. adapting mixing_beta), or doing an "incorrect stop".

I was curious if you could "optimize" the choice for electron_maxstep, i.e. minimize the % of incorrect stops. But to judge this properly I think we need to have data for larger electron_maxstep runs, since we don't really know which of the failed runs would have converged if we had run with more than 80 steps. Also: we don't know if switching to e.g. a lower mixing_beta would improve or worsen the convergence after step 50. So perhaps we shouldn't overdo this analysis for now.

Maybe we should compute the slope only on e.g. 20 of the last steps

I was also thinking of this approach. Here is the analysis:

$ ./gather_stats.py slopes --last 20 --max-nstep 50 workchain/PBEsol/scf
Report: Number of completed pw.x with > 50 SCF steps: 3524
Report: Calculating slopes from step 30 to 50.
Report: Incorrect stops: 130 | 12.7% of 1026 successful runs with > 50 steps
Report: Incorrect restarts: 834 | 33.4% of 2498 failed runs

So based on this setup (information about 80 steps, trial "max nstep" 50, slope threshold -0.1), only checking the last 20 steps increases the # of incorrect stops, but reduces the # of incorrect restarts. Below are some examples of runs where we would incorrectly stop:

So not sure if this really improves things. I suppose it depends what we prioritise: convergence or saving resources.

Thanks @mbercx ! Ok, I see now. Well, the the point is also whether a change of strategy (e.g. mixing beta) will ever produce significant change in the first 50 (or 60, or whatever) iterations. There is a high chance to keep trying changing strategy, while having just some iterations more would solve the convergence.
So I don't know how much we are going to save resources if we are not really sure to fix with a certain probability the problem. So I would vote to even increase such maximum number of iterations (which btw on QE I think the default is to 100).

So if we start from 0.4 then the next will be: 0.2, 0.1, 0.05, ... Nevertheless, 0.05 is rather uncommon. Shall we set something to reproduce e.g. 0.3, 0.2, 0.1, 0.05? Or probably just set a list of them? Ok, here I don't consider starting from a different mixing beta. What do you thik?

Actually coming back here - probably it is just best to leave 0.5 as a factor, son one explores quickly the possible betas. No need to stay too close I guess.

Frankly, I'm not sure either. My thinking is that in case reducing the mixing improves the convergence, better to reduce a bit more and then check if the slope is < -0.1 after 50 steps and then give it more max steps.

Yeah makes sense, something like in the ph.x code, where you can chnage mixing beta at a certain point

max_electronic_steps too? E.g. ~200-250?

I would also take the intercept to use for extrapolating the amount of extra steps to perform. Or otherwise, we can simply increase it to a fixed maximum? E.g. electrons_maxstep = self.defaults.max_electronic_steps?

Update ELECTRONS.electron_maxstep. Discussion needed

mixing_mode = self.ctx.inputs.parameters['ELECTRONS'].get_default('mixing_beta', 'plain')
if mixing_mode == 'plain':
  self.ctx.inputs.parameters['ELECTRONS']['mixing_mode'] = 'local-TF'
  [...]
mixing_ndim = self.ctx.inputs.parameters['ELECTRONS'].get_default('mixing_ndim', 8)
if mixing_ndim <= 8:
  self.ctx.inputs.parameters['ELECTRONS']['mixing_ndim'] = 20 
  [...]

I would try also these solutions. Maybe at the same time?