Back testing and parameters optimization

A great opportunity to quickly and safely check how changing parameters affects the result

Concept

Modified Martingale strategy responds to market signals, constantly adjusting to changing conditions. The system itself is passive - it only responds to an incoming data stream. The reaction depends on:

1. Specified limits of parameters change
2. System status at a given time
3. Incoming data from exchange:
   • market data stream
   • user data stream
4. Time scale, some process have fixed or parametric time delay

For quality simulation and obtaining a repeatable result, all of the above matters.

The following solutions have been implemented for these items:

1. This is the subject of research and optimization, with identical parameters, the strategy gives the same result in back testing mode
2. Data collection and training sessions start from a zero state. The ability to start from any saved state is being studied
3. Exchange data emulator
   • market data - during the TC (Trade & Collect) session, the required set of data streams is stored
   • user data - developed an exchange emulator that reliably reproduces basic functions, such as placing and executing orders and updating the accounting balance
4. Playback of the data stream is accelerated xN times, the "local time generator" is synchronized with this stream, all internal processes of the system are synchronously accelerated

Limitations

• Only the main functions of the exchange are emulated, except for the above ones, time delays when placing orders are taken into account. However, this is a simple model that does not implement partial order fulfillment, market liquidity, slippage, the impact of own orders on the order book, etc. It should be taken into account that market conditions are changing in real time and super productive parameters in the past period may not be so effective in the future.

• Use Trade & Collect mode on product system with caution. The accumulation of streaming data requires additional memory. There is a safety mechanism - if the amount of free memory in the system becomes less than 30%, data from the memory is uploaded to the file and the memory occupied for this process is freed.

Collect mode

New parameter group added to configuration file:

# Backtest mode parameters
ex.MODE = 'T'  # 'T' - Trade, 'TC' - Trade and Collect, 'S' - Simulate
ex.SAVE_DS = True  # Save session result data (ticker, orders) for compare
ex.SAVE_PERIOD = 24 * 60 * 60  # sec, timetable for save data portion, but memory limitation consider also matter
ex.SAVED_STATE = False  # Use saved state for backtesing

Use ex.MODE = 'TC' for collect WS stream data for later back-testing. You can use this mode for both - "paper" and real trading.

Set ex.SAVED_STATE = True if you want optimize started strategy. In this case it is necessary:

• Stop strategy with save current state
• Set Back testing parameters
• Restart strategy from saved state in Trade&Collect mode

As result, after stop strategy new file structure was created:

Where:

• back_test/binance_BTCUSDT/raw - saved WS data for later back-testing
• back_test/binance_BTCUSDT/snapshot - snapshot of the current session, containing the order history, separately buy/sell and ticker
• back_test/binance_BTCUSDT/cli_7_BTCUSDT.py - copy of session parameters
• back_test/binance_BTCUSDT-HH:MM.zip - archive with unloaded portion of session data

The data is uploaded to the archive by an event that occurs earlier:

• stop strategy by command
• recording time exceeded, setting by parameter SAVE_PERIOD = 24 * 60 * 60 # sec, timetable for save data portion. Default value 24h.
• free memory in the system has become less than 30%

Simulate mode

Take saved back_test/binance_ETHBUSD/cli_7_ETHBUSD.py, set ex.MODE = 'S' and start it.

For the first time, do not change the trading parameters, so you can make sure that the result is reliably obtained. For comparison you can use visual method

The trading simulation will start without further confirmation. A real connection to the exchange will be used to obtain initial data to initialize the strategy. In this mode, real orders are not sent to the exchange.

After ending of data set, common result wold be displayed:

Backtest candles *** 15m *** timeSeries ended
Backtest candles *** 1h *** timeSeries ended
Backtest candles *** 1m *** timeSeries ended
20:02:09.606230 Backtest *** ticker *** timeSeries ended
Original time: 0:08:56.241000, test time: 0:00:01.817368, x = 295.06
Session data saved to: /home/ubuntu/.MartinBinance/back_test/binance_ETHBUSD_0609-20:02:41
Session profit: 0E-7, free: 0.0183900, total: 0.01839
20:02:09.615905 Got signal for exit

And session data snapshot can be found at the specified location.

By changing the trading parameters, you can get a different result. It's not very productive, but it's quite entertaining. For efficient parameter matching, use Parameters optimization solution.

Visual comparison of session shots

Run martin_binance/backtest/VCoSEL.py. Select path for sessions snapshot for compare:

• snapshot of the original session back_test/binance_ETHBUSD/snapshot/
• snapshot of testing session back_test/binance_ETHBUSD_mmdd-HH:MM/

View result at http://127.0.0.1:8050/

Parameters optimization

To find the optimal trading parameters, the framework optuna is used.

Look at the file martin_binance/backtest/OoTSP.py. Here the parameters for optimization and the limits of their change are set.

def objective(trial):
    params = {
        'GRID_MAX_COUNT': trial.suggest_int('GRID_MAX_COUNT', 3, 5),
        'PRICE_SHIFT': trial.suggest_float('PRICE_SHIFT', 0, 0.05, step=0.01),
        'PROFIT': trial.suggest_float('PROFIT', 0.05, 0.15, step=0.05),
        'PROFIT_MAX': trial.suggest_float('PROFIT_MAX', 0.25, 1.0, step=0.05),
        'OVER_PRICE': trial.suggest_float('OVER_PRICE', 0.1, 1, step=0.1),
        'ORDER_Q': trial.suggest_int('ORDER_Q', 6, 12),
        'MARTIN': trial.suggest_float('MARTIN', 5, 15, step=1),
        'SHIFT_GRID_DELAY': trial.suggest_int('SHIFT_GRID_DELAY', 10, 60, step=5),
        'KBB': trial.suggest_float('KBB', 1, 5, step=0.5),
        'LINEAR_GRID_K': trial.suggest_int('LINEAR_GRID_K', 0, 100, step=10),
    }
    return try_trade(**params)

Just run the

martin-binance-backtest

in terminal window and select optimization parameters:

ubuntu@vm2:~$ martin-binance-backtest
[?] Select from saved: exchange_PAIR with the strategy you want to optimize: okx_OKBUSDC
 ❯ okx_OKBUSDC

[?] New study session or optimization history plot from saved one: New
 ❯ New
   Plot from saved

[?] Enter number of cycles, from 50 to 500: 2
[I 2023-06-21 15:24:46,352] A new study created in RDB with name: okx_OKBUSDC

After end of study cycle you get optimization report:

[I 2023-06-21 15:31:29,902] Trial 1 finished with value: 3.961858 and parameters: {'GRID_MAX_COUNT': 3, 'KBB': 2.0, 'LINEAR_GRID_K': 100, 'MARTIN': 9.0, 'ORDER_Q': 11, 'OVER_PRICE': 0.1, 'PRICE_SHIFT': 0.02, 'PROFIT': 0.15, 'PROFIT_MAX': 0.7, 'SHIFT_GRID_DELAY': 50}. Best is trial 1 with value: 3.961858.
Optimal parameters: {'GRID_MAX_COUNT': 3, 'KBB': 2.0, 'LINEAR_GRID_K': 100, 'MARTIN': 9.0, 'ORDER_Q': 11, 'OVER_PRICE': 0.1, 'PRICE_SHIFT': 0.02, 'PROFIT': 0.15, 'PROFIT_MAX': 0.7, 'SHIFT_GRID_DELAY': 50} for get 3.961858
Evaluate parameter importance based on completed trials in the given study:
('OVER_PRICE', 0.17142857142866355)
('KBB', 0.14285714285712012)
('SHIFT_GRID_DELAY', 0.11428571428577569)
('ORDER_Q', 0.11428571428569612)
('GRID_MAX_COUNT', 0.11428571428569609)
('LINEAR_GRID_K', 0.08571428571433179)
('PROFIT_MAX', 0.08571428571427207)
('PROFIT', 0.08571428571421236)
('PRICE_SHIFT', 0.057142857142808236)
('MARTIN', 0.028571428571424022)
Study instance saved to sqlite:////home/ubuntu/.MartinBinance/back_test/okx_OKBUSDC/okx_OKBUSDC.db for later use

If you have not GUI on production instance, you can copy Study instance ~/.MartinBinance/back_test/okx_OKBUSDC/okx_OKBUSDC.db to GUI environment for for the study. Use mode "Plot from saved" and read optuna docs.