OfferCreate with tfFillOrKill fails with tecKILLED if offer is better than, but doesn't exactly match, open offer rate

[nhartner]

Issue Description

CreateOffer with tfFillOrKill flag will always results in a tecKILLED result if the exchange rate on the offer is better than, but doesn't exactly match, the order book rate.

Steps to Reproduce

1. Create 3 accounts and fund them with faucet:

• ISSUER (ex. https://test.bithomp.com/explorer/r1MfR17bEHV1VhFgfZey6TizQKueEJwLV)
• MAKER (ex. https://test.bithomp.com/explorer/rML3sUiiYUnMj7WPc9ECNdMXnYe5Z5Xbdb)
• TAKER (ex. https://test.bithomp.com/explorer/rSeyhj86z7GCQ96EV3X8naQA6V57VQoN9)

2. Enable Default Ripple flag on ISSUER (ex. https://test.bithomp.com/explorer/424CF9443FBBE23734DF9E49BA2297D2A2ECE58C11F01B4EE8FD251F63E85A6C)

3. Create trustlines:

• From MAKER to ISSUER for currency ONE (ex. https://test.bithomp.com/explorer/9F03819996B0289137B0BC9297DEDA53C9DEDD4DA99531989719BAD5DC2CC776)
• From TAKER to ISSUER for currency ONE (ex. https://test.bithomp.com/explorer/72EB305366A3897BF70BB37F1ECF5065A5D4DC48B3D244569B68786242D02F33)

4. Send a payment of 100 units of currency ONE from ISSUER to MAKER (ex. https://test.bithomp.com/explorer/BBFBC0CB6B6ECE68050E8C57E7F7F33F0437D9EF03E638F74D70DFF5DBF1B06B)

5. Create an offer on MAKER account to sell 100 units of currency ONE for 100 XRP (ex. https://test.bithomp.com/explorer/DDFF0C28494332C491147A9A85AC8F5C1A988A848B4366271E72A67266AF5556)

6. Create an offer on TAKER account to buy 100 units of currency ONE for 101 XRP with tfFillOrKill flag (ex. https://test.bithomp.com/explorer/7251BC0895848886B9419FC1E209DABAAA2BE3BD66CB690BABFD92D660EA4743)

Expected Result

Offer should fill and not be killed because TAKER is offering to pay 101 XRP for 100 ONE and maker is offering to sell 100 ONE for 100 XRP. Since TAKER is offering MORE XRP than the MAKER is asking, the order should match and execute with a tesSUCCESS result.

Actual Result

Offer is killed with tecKILLED status. (see https://test.bithomp.com/explorer/7251BC0895848886B9419FC1E209DABAAA2BE3BD66CB690BABFD92D660EA4743)

Note:

• If the TAKER offers to pay exactly 100 XRP for 100 ONE, the tfFillOrKill offer will get fully matched
• If the tfImmediateOrCancel flag is used instead, the offer for 101 XRP for 100 ONE also gets fully matched

Environment

Reproduced on testnet and mainnet

[scottschurr]

@gregtatcam has a unit test reproducing what @nhartner is observing. It looks like the problem was introduced with the flowCross amendment which went live August of 2020. @gregtatcam has offered to write an amendment with a fix.