Back testing and parameters optimization

A great opportunity to quickly and safely check how changing parameters affects the result

Concept

Modified Martingale strategy responds to market signals, constantly adjusting to changing conditions. The system itself is passive - it only responds to an incoming data stream. The reaction depends on:

1. Specified limits of parameters change
2. System status at a given time
3. Incoming data from exchange:
   • market data stream
   • user data stream
4. Time scale, some process have fixed or parametric time delay

For quality simulation and obtaining a repeatable result, all of the above matters.

The following solutions have been implemented for these items:

Exchange data emulator

• market data - during the TC (Trade & Collect) session, the required set of data streams is stored
• user data - developed an exchange emulator that reliably reproduces basic functions, such as placing and executing orders and updating the accounting balance

4. Playback of the data stream is accelerated xN times, the "local time generator" is synchronized with this stream, all internal processes of the system are synchronously accelerated

Limitations

• Only the main functions of the exchange are emulated, except for the above ones, time delays when placing orders are taken into account. However, this is a simple model that does not implement partial order fulfillment, market liquidity, slippage, the impact of own orders on the order book, etc. It should be taken into account that market conditions are changing in real time and super productive parameters in the past period may not be so effective in the future.

Collect mode

New parameter group added to configuration file:

ex.MODE = 'TC'  # 'T' - Trade, 'TC' - Trade and Collect, 'S' - Simulate
ex.SAVE_DS = False  # Save session result data (ticker, orders) for compare
ex.SAVE_PERIOD = 3 * 60 * 60  # sec, timetable for save data portion
ex.SELF_OPTIMIZATION = True  # Cyclic self-optimization of parameters, together with MODE == 'TC'
ex.N_TRIALS = 500  # Number of optimization cycles for optuna study in self optimization mode

Use ex.MODE = 'TC' for collect WS stream data for later back-testing. You can use this mode for both - "paper" and real trading.

• Stop strategy with save current state
• Set Back testing parameters
• Restart strategy from saved state in Trade&Collect mode

As result, after SAVE_PERIOD time you get message:

16/02 20:19:34 Stream data for backtesting saved to /home/ubuntu/.MartinBinance/back_test/bybit_BTCUSDT
16/02 20:19:34 Backtest data collect and optimize session ended

and new file structure was created:

Where:

• back_test/bybit_BTCUSDT/raw - saved WS data for later back-testing
• back_test/bybit_BTCUSDT/snapshot - snapshot of the current session, containing the order history, separately buy/sell
• back_test/bybit_BTCUSDT/cli_36_BTCUSDT.py - copy of session parameters
• back_test/bybit_BTCUSDT/saved_state.json - saved state before start collecting

Simulate mode

Take saved back_test/binance_ETHBUSD/cli_7_ETHBUSD.py, set ex.MODE = 'S' and start it.

For the first time, do not change the trading parameters, so you can make sure that the result is reliably obtained. For comparison you can use visual method

The trading simulation will start without further confirmation. A real connection to the exchange will be used to obtain initial data to initialize the strategy. In this mode, real orders are not sent to the exchange.

After ending of data set, common result wold be displayed:

Backtest candles *** 15m *** timeSeries ended
Backtest candles *** 1h *** timeSeries ended
Backtest candles *** 1m *** timeSeries ended
20:02:09.606230 Backtest *** ticker *** timeSeries ended
Original time: 0:08:56.241000, test time: 0:00:01.817368, x = 295.06
Session data saved to: /home/ubuntu/.MartinBinance/back_test/binance_ETHBUSD_0609-20:02:41
Session profit: 0E-7, free: 0.0183900, total: 0.01839
20:02:09.615905 Got signal for exit

And session data snapshot can be found at the specified location.

By changing the trading parameters, you can get a different result. It's not very productive, but it's quite entertaining. For efficient parameter matching, use Parameters optimization solution.

Visual comparison of session shots

Run martin_binance/backtest/VCoSEL.py. Select path for sessions snapshot for compare:

• snapshot of the original session back_test/bybit_BTCUSDT/snapshot/
• snapshot of testing session back_test/bybit_BTCUSDT_mmdd-HH-MM-SS/

View result at http://127.0.0.1:8050/

Parameters optimization

To find the optimal trading parameters, the framework optuna is used.

🚀 Starting with version 2.1.2, new functionality has become: continuous cyclic search for optimal meta-parameters

Just set parameters:

ex.MODE = 'TC'  # 'T' - Trade, 'TC' - Trade and Collect, 'S' - Simulate
ex.SELF_OPTIMIZATION = True  # Cyclic self-optimization of parameters, together with MODE == 'TC'
ex.SAVE_PERIOD = 6 * 60 * 60  # sec, timetable for save data portion
ex.N_TRIALS = 500  # Number of optimization cycles for optuna study in self optimization mode

and restart strategy. Periodically you get message from back testing control subsystem*:

17/02 18:09:25 Updating strategy parameters from backtest optimal, predicted value 34.30023114 -> 72.85014514
17/02 18:09:25 GRID_MAX_COUNT: 5 -> 3
17/02 18:09:25 PRICE_SHIFT: 0 -> 0.0
17/02 18:09:25 PROFIT: 0.1 -> 0.2
17/02 18:09:25 PROFIT_MAX: 0.35 -> 0.9
17/02 18:09:25 OVER_PRICE: 0.6 -> 0.1
17/02 18:09:25 ORDER_Q: 12 -> 10
17/02 18:09:25 MARTIN: 10 -> 5.0
17/02 18:09:25 SHIFT_GRID_DELAY: 5 -> 110
17/02 18:09:25 KBB: 0.1 -> 1.5
17/02 18:09:25 LINEAR_GRID_K: 0 -> 80
17/02 18:09:25 Strategy parameters are optimal now. Optimization cycle duration 9:13:32

*After the first optimization stage, the parameters will be updated in any case, even if the income of the period is zero.

Some details:

You can arbitrarily change the composition of parameters and limit values. Edit the .MartinBinance/trial_params.json file. There is no need to restart the strategy, updates will be used in the next cycle. This setting is the same for all trading pairs.

{
  "GRID_MAX_COUNT": {"type": "int", "range": [3, 5]},
  "PRICE_SHIFT": {"type": "float", "range": [0, 0.05], "step": 0.01},
  "PROFIT": {"type": "float", "range": [0.05, 0.2], "step": 0.05},
  "PROFIT_MAX": {"type": "float", "range": [0.4, 1.0], "step": 0.05},
  "OVER_PRICE": {"type": "float", "range": [0.1, 1], "step": 0.1},
  "ORDER_Q": {"type": "int", "range": [6, 12]},
  "MARTIN": {"type": "float", "range": [5, 15], "step": 0.5},
  "SHIFT_GRID_DELAY": {"type": "int", "range": [10, 150], "step": 10},
  "KBB": {"type": "float", "range": [0.5, 4], "step": 0.5},
  "LINEAR_GRID_K": {"type": "int", "range": [0, 500], "step": 50}
}

When setting limit values, consider the type and step ratio. Type can be int or float, float matches Decimal() in cli_X_AAABBB.py.

Just run the

martin-binance-backtest

in terminal window and select optimization parameters:

ubuntu@vm2:~$ martin-binance-backtest
[?] Select from saved: exchange_PAIR with the strategy you want to optimize: bybit_BTCUSDT
 ❯ bybit_BTCUSDT

[?] New study session or analise from saved one: New
 ❯ New
   Analise saved study session

[?] Enter number of cycles, from 10 to 1000: 50

After end of study cycle you get optimization report:

Best trial: 5. Best value: 0.746675: 100%|███████████████████████████████| 50/50 [04:42<00:00,  5.65s/it]
Optimal parameters: {'GRID_MAX_COUNT': 4, 'PRICE_SHIFT': 0.04, 'PROFIT': 0.05, 'PROFIT_MAX': 0.9, 'OVER_PRICE': 0.30000000000000004, 'ORDER_Q': 10, 'MARTIN': 5.0, 'SHIFT_GRID_DELAY': 80, 'KBB': 1.0, 'LINEAR_GRID_K': 0} for get 0.74667511
Evaluate parameter importance based on completed trials in the given study:
('PROFIT_MAX', 0.3246408024767296)
('KBB', 0.265048501896488)
('MARTIN', 0.11786167668624496)
('LINEAR_GRID_K', 0.07893770198345028)
('OVER_PRICE', 0.07800826498352383)
('SHIFT_GRID_DELAY', 0.0705445672491495)
('PRICE_SHIFT', 0.030134064415064766)
('PROFIT', 0.022059266650531328)
('ORDER_Q', 0.010676851027290957)
('GRID_MAX_COUNT', 0.0020883026315269555)
Study instance saved to sqlite:////home/ubuntu/.MartinBinance/back_test/bybit_BTCUSDT/study.db for later use

If you have not GUI on production instance, you can copy /back_test/bybit_BTCUSDT/study.db to GUI environment for the study. Use mode "Plot from saved" and read optuna docs.