Add metric and retry backoff for checksum failure

[Shaddoll]

What changed?
Add metric and retry backoff for checksum failure

Why?

• backoff a small period of time to give some time for the database to get refresh data
• add a metric to track the number of stale reads which aren't mitigated by the retry logic

How did you test it?

Potential risks

Release notes

Documentation Changes

[coveralls]

Pull Request Test Coverage Report for Build 018e57be-d364-4a21-baf7-abad984c73ee

Details

• 4 of 7 (57.14%) changed or added relevant lines in 1 file are covered.
• 15 unchanged lines in 7 files lost coverage.
• Overall coverage increased (+0.03%) to 65.119%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
service/history/execution/context.go	4	7	57.14%

Files with Coverage Reduction	New Missed Lines	%
service/history/execution/context.go	1	68.08%
service/history/task/transfer_standby_task_executor.go	2	87.01%
common/task/weighted_round_robin_task_scheduler.go	2	89.05%
common/task/fifo_task_scheduler.go	2	87.63%
service/history/execution/mutable_state_builder.go	2	70.3%
service/history/queue/timer_queue_processor_base.go	3	77.82%
service/history/queue/timer_gate.go	3	95.83%

Totals
Change from base Build 018e57d2-73cf-4c4c-be3d-a60180509d80:	0.03%
Covered Lines:	95182
Relevant Lines:	146167

---

💛 - Coveralls