Back testing and parameters optimization

Under maintenance

A great opportunity to quickly and safely check how changing parameters affects the result

Concept

Modified Martingale strategy responds to market signals, constantly adjusting to changing conditions. The system itself is passive - it only responds to an incoming data stream. The reaction depends on:

1. Specified limits of parameters change
2. System status at a given time
3. Incoming data from exchange:
   • market data stream
   • user data stream
4. Time scale, some process have fixed or parametric time delay

For quality simulation and obtaining a repeatable result, all of the above matters.

The following solutions have been implemented for these items:

Exchange data emulator

• market data - during the TC (Trade & Collect) session, the required set of data streams is stored
• user data - developed an exchange emulator that reliably reproduces basic functions, such as placing and executing orders and updating the accounting balance

4. Playback of the data stream is accelerated xN times, the "local time generator" is synchronized with this stream, all internal processes of the system are synchronously accelerated

Limitations

• Only the main functions of the exchange are emulated, except for the above ones, time delays when placing orders are taken into account. However, this is a simple model that does not implement partial order fulfillment, market liquidity, slippage, the impact of own orders on the order book, etc. It should be taken into account that market conditions are changing in real time and super productive parameters in the past period may not be so effective in the future.

Collect mode

New parameter group added to configuration file:

ex.MODE = 'TC'  # 'T' - Trade, 'TC' - Trade and Collect, 'S' - Simulate
ex.SAVE_DS = False  # Save session result data (ticker, orders) for compare
ex.SAVE_PERIOD = 3 * 60 * 60  # sec, timetable for save data portion
ex.SELF_OPTIMIZATION = True  # Cyclic self-optimization of parameters, together with MODE == 'TC'
ex.N_TRIALS = 500  # Number of optimization cycles for optuna study in self optimization mode

Use ex.MODE = 'TC' for collect WS stream data for later back-testing. You can use this mode for both - "paper" and real trading.

• Stop strategy with save current state
• Set Back testing parameters
• Restart strategy from saved state in Trade&Collect mode

As result, after SAVE_PERIOD time you get message:

16/02 20:19:34 Stream data for backtesting saved to /home/ubuntu/.MartinBinance/back_test/bybit_BTCUSDT
16/02 20:19:34 Backtest data collect and optimize session ended

and new file structure was created:

Where:

• back_test/bybit_BTCUSDT/raw - saved WS data for later back-testing
• back_test/bybit_BTCUSDT/snapshot - snapshot of the current session, containing the order history, separately buy/sell
• back_test/bybit_BTCUSDT/cli_36_BTCUSDT.py - copy of session parameters
• back_test/bybit_BTCUSDT/saved_state.json - saved state before start collecting

Simulate mode

Take saved back_test/binance_ETHBUSD/cli_7_ETHBUSD.py, set ex.MODE = 'S' and start it.

For the first time, do not change the trading parameters, so you can make sure that the result is reliably obtained. For comparison you can use visual method

The trading simulation will start without further confirmation. A real connection to the exchange will be used to obtain initial data to initialize the strategy. In this mode, real orders are not sent to the exchange.

After ending of data set, common result wold be displayed:

Backtest candles *** 15m *** timeSeries ended
Backtest candles *** 1h *** timeSeries ended
Backtest candles *** 1m *** timeSeries ended
20:02:09.606230 Backtest *** ticker *** timeSeries ended
Original time: 0:08:56.241000, test time: 0:00:01.817368, x = 295.06
Session data saved to: /home/ubuntu/.MartinBinance/back_test/binance_ETHBUSD_0609-20:02:41
Session profit: 0E-7, free: 0.0183900, total: 0.01839
20:02:09.615905 Got signal for exit

And session data snapshot can be found at the specified location.

By changing the trading parameters, you can get a different result. It's not very productive, but it's quite entertaining. For efficient parameter matching, use Parameters optimization solution.

Visual comparison of session shots

Run martin_binance/backtest/VCoSEL.py. Select path for sessions snapshot for compare:

• snapshot of the original session back_test/bybit_BTCUSDT/snapshot/
• snapshot of testing session back_test/bybit_BTCUSDT_mmdd-HH-MM-SS/

View result at http://127.0.0.1:8050/

Parameters optimization

To find the optimal trading parameters, the framework optuna is used.

Look at the file martin_binance/backtest/OoTSP.py. Here the parameters for optimization and the limits of their change are set.

def objective(trial):
    params = {
        'GRID_MAX_COUNT': trial.suggest_int('GRID_MAX_COUNT', 3, 5),
        'PRICE_SHIFT': trial.suggest_float('PRICE_SHIFT', 0, 0.05, step=0.01),
        'PROFIT': trial.suggest_float('PROFIT', 0.05, 0.15, step=0.05),
        'PROFIT_MAX': trial.suggest_float('PROFIT_MAX', 0.25, 1.0, step=0.05),
        'OVER_PRICE': trial.suggest_float('OVER_PRICE', 0.1, 1, step=0.1),
        'ORDER_Q': trial.suggest_int('ORDER_Q', 6, 12),
        'MARTIN': trial.suggest_float('MARTIN', 5, 15, step=1),
        'SHIFT_GRID_DELAY': trial.suggest_int('SHIFT_GRID_DELAY', 10, 60, step=5),
        'KBB': trial.suggest_float('KBB', 1, 5, step=0.5),
        'LINEAR_GRID_K': trial.suggest_int('LINEAR_GRID_K', 0, 100, step=10),
    }
    return try_trade(**params)

Just run the

martin-binance-backtest

in terminal window and select optimization parameters:

ubuntu@vm2:~$ martin-binance-backtest
[?] Select from saved: exchange_PAIR with the strategy you want to optimize: okx_OKBUSDC
 ❯ okx_OKBUSDC

[?] New study session or optimization history plot from saved one: New
 ❯ New
   Plot from saved

[?] Enter number of cycles, from 50 to 500: 2
[I 2023-06-21 15:24:46,352] A new study created in RDB with name: okx_OKBUSDC

After end of study cycle you get optimization report:

[I 2023-06-21 15:31:29,902] Trial 1 finished with value: 3.961858 and parameters: {'GRID_MAX_COUNT': 3, 'KBB': 2.0, 'LINEAR_GRID_K': 100, 'MARTIN': 9.0, 'ORDER_Q': 11, 'OVER_PRICE': 0.1, 'PRICE_SHIFT': 0.02, 'PROFIT': 0.15, 'PROFIT_MAX': 0.7, 'SHIFT_GRID_DELAY': 50}. Best is trial 1 with value: 3.961858.
Optimal parameters: {'GRID_MAX_COUNT': 3, 'KBB': 2.0, 'LINEAR_GRID_K': 100, 'MARTIN': 9.0, 'ORDER_Q': 11, 'OVER_PRICE': 0.1, 'PRICE_SHIFT': 0.02, 'PROFIT': 0.15, 'PROFIT_MAX': 0.7, 'SHIFT_GRID_DELAY': 50} for get 3.961858
Evaluate parameter importance based on completed trials in the given study:
('OVER_PRICE', 0.17142857142866355)
('KBB', 0.14285714285712012)
('SHIFT_GRID_DELAY', 0.11428571428577569)
('ORDER_Q', 0.11428571428569612)
('GRID_MAX_COUNT', 0.11428571428569609)
('LINEAR_GRID_K', 0.08571428571433179)
('PROFIT_MAX', 0.08571428571427207)
('PROFIT', 0.08571428571421236)
('PRICE_SHIFT', 0.057142857142808236)
('MARTIN', 0.028571428571424022)
Study instance saved to sqlite:////home/ubuntu/.MartinBinance/back_test/okx_OKBUSDC/okx_OKBUSDC.db for later use

If you have not GUI on production instance, you can copy Study instance ~/.MartinBinance/back_test/okx_OKBUSDC/okx_OKBUSDC.db to GUI environment for the study. Use mode "Plot from saved" and read optuna docs.