Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File-based options universe #8212

Merged

Conversation

jhonabreul
Copy link
Collaborator

@jhonabreul jhonabreul commented Jul 18, 2024

Description

Refactor Equity and Index Option universe selection:

  • Now file-based: option chains are directly read from files instead of being built from price data file names.
  • Option contracts include more data at selection time, which means they can be filtered by their data:
    • Data includes Greeks, Implied Volatility and Open Interest.
    • New filters where added to allow this.
  • History requests can now be used to fetch this options data, which would be like getting a full unfiltered option chain with the new additional data.

Notes about some algorithms with changed stats:

  • Index BasicTemplateSPXWeeklyIndexOptionsAlgorithm:
    • Stats change because since options universe data is daily, an additional Sample() calls is made on 2021/1/9 midnight
  • IndexOptionScaledStrikeRegressionAlgorithm:
    • Orders and stats change because now selection is immediate, so orders are place for the exact same options but an hour earlier, and price is different at that time.
  • BasicTemplateOptionsFrameworkAlgorithm:
    • Same contracts traded, just at different time of day with a different price

Related Issue

Part of #6861

Motivation and Context

  • Performance improvements for options algorithms.
  • Greeks and other data at selection time, which allows improved filtering.

Requires Documentation Change

New filters should be documented

How Has This Been Tested?

  • Exisiting regression algorithms and unit tests.
  • New regression algorithms and unit tests.
  • Cloud tests:
    • Sample options algorithms
    • Live deployments running thought multiple days with a bunch of logs:
      • Look that first selection happens right after deployment.
      • Look that selection happens everyday after data is available.
Backtesting benchmarks
1. Basic speed test:

Algorithm with a span of 4 years adding SPX options with minute resolution:

Case 1: Selecting 0 contracts in the filter function

This case allows us to test the chain loading and selection speed, without price data.

Case 2: Selecting up to 10 contracts each date

Not that big of a difference once contracts are selected, but pure contracts loading is much faster.

2. Filtering speed test:

SPY options algorithm of 6 month with daily resolution, doing some work in the filter function to get implied volatility values for filtering:

  • Master: fetching historical data in the filter function, feeding it to an ImpliedVolatility indicator and using the resulting IV values to do filtering.
  • Branch: using pre-calculated implied volatility in OptionUniverse instances.
Case 1: Selecting all contracts with IV in [50% IV Median, 150% IV Median]

A time reduction of about 30%.

Case 2: Doing the IV calculations, but selecting 0 contracts.

This case allows us to test speed difference for option contracts loading and filtering only, without price data fetching and loading taken into consideration.

Selection based on greeks is now dramatically faster and simpler.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which improves implementation)
  • Performance (non-breaking change which improves performance. Please add associated performance test and results)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Non-functional change (xml comments/documentation/etc)

Checklist:

  • My code follows the code style of this project.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

Copy link
Member

@Martin-Molinero Martin-Molinero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Nice work, leaving a few minor comments 👍

Algorithm.CSharp/BasicTemplateOptionsFilterAlgorithm.cs Outdated Show resolved Hide resolved
Common/Data/UniverseSelection/OptionChainUniverse.cs Outdated Show resolved Hide resolved
Common/Data/UniverseSelection/OptionChainUniverse.cs Outdated Show resolved Hide resolved
Research/QuantBook.cs Show resolved Hide resolved
Engine/DataFeeds/DataManager.cs Show resolved Hide resolved
Common/Securities/IDerivativeSecurityFilterUniverse.cs Outdated Show resolved Hide resolved
@jhonabreul jhonabreul force-pushed the feature-options-greeks-data branch 3 times, most recently from cfdcb22 to ad0bd54 Compare July 30, 2024 15:31
@jhonabreul jhonabreul force-pushed the feature-options-greeks-data branch from 166aac0 to 9371f6a Compare July 30, 2024 20:14
@jhonabreul jhonabreul force-pushed the feature-options-greeks-data branch 3 times, most recently from f46ef9f to 3f716ba Compare August 19, 2024 15:32
@jhonabreul jhonabreul force-pushed the feature-options-greeks-data branch from 201aa70 to c863347 Compare August 26, 2024 16:47
@jhonabreul jhonabreul force-pushed the feature-options-greeks-data branch from b78d2b4 to 3327576 Compare September 4, 2024 20:45
@Martin-Molinero Martin-Molinero merged commit e29bb2c into QuantConnect:master Sep 9, 2024
10 of 14 checks passed
@jhonabreul jhonabreul deleted the feature-options-greeks-data branch September 9, 2024 15:47
wtindall1 pushed a commit to wtindall1/Lean that referenced this pull request Nov 10, 2024
* Initial options universe with greeks implementation

* Options universe improvements

* Address peer review

* File based options universe fixes and improvements.

- Adjust OptionUniverse start-end times and period.
- Adapt unit tests and some algorithms to pass with new options universe selection.

* Updated options regression algorithms stats for new universe data

* Updated options regression algorithms stats for new universe data

* Updated options regression algorithms stats for new universe data

* Updated options regression algorithms stats for new universe data

* Updated options regression algorithms stats for new universe data

* Option chain provider with new options universe

* Allow canonical option history requests

* Address peer review

* Address peer review

* Fix symbols parsing in OptionUniverse

* Fix universe selection subscriptions start time to not include extended market hours

* Minor changes

* Minor changes

* Peer recommended changes and fixes

* Update regression algorithm stats

* Update regression algorithms stats and minor fixes

* Fix option chain provider history request

* Round option indicators values

* Added option universe csv header property

* Update regression algorithms stats

* Update regression algorithms stats

* Data fixes and regression algos stats update

* Unit test fixes

* Minor changes

* Option chain handling in live trading data feed

* Minor changes

* Added processed data provider

* Fix thread-safety violation in Slice class

* Minor change

* Update options filter universe API to use OptionUniverse data

Add new filter methods for greeks, IV and open interest

* Option filter universe api updates

* Add OptionUniverse history regression algorithms

* Add regression algorithms for new options filter universe api methods

* Added options greeks data and updated regression algorithms

* Address peer review

* Address peer review

* Add more assertions to new options filter api regression algorithms

* Minor performance improvement.

Reduce greeks binomial model steps to 140

* Minor tests updates

* Greeks numerical models performance improvements

* Greeks numerical models performance improvements

* Revert array pool change for option pricing numerical models

* Update default dividend yield provider depending on option type

* [TEST]

* Add helper method con calculate time till expiration

* Use double in price option numerical models

* Implied volatility calculation improvements

- Adjust root finding method accuracy as a factor of the option price
- Use BSM to get a first guess

* Cleanup

* Some regression algorithms and unit tests cleanup

* Regression tests updates after rebasing from master

* Add universe files

* Self review and cleanup

* Minor regression tests updates after rebase

* Fix: set data time zone to same as exchange tz for options universes

* Minor change

* Minor change

* Fix for live trading options universe selection

* Keep underlying when aggregating collections in BaseDataCollectionAggregatorEnumerator

* Update index options regression algorithms stats

* Minor change

* Address peer review

* Memory usage improvements

* Minor build fix

* Minor changes and test fixes

* Cache symbols in OptionUniverse

* Cleanup

* Fix index option creation in OptionUniverse

* Use cached underlying SID when parsing from string

* Abstract symbols cache to BaseDataCollection

* Return actual underlying symbol when mapping decomposing ICO ticker

* Address peer review

* Minor performance improvements reduce garbage

* Limit Symbols and SIDs cache size to help with memory usage

* Minor fix in symbols and sid cache cleanup

* Build fix

* Lazily parse greeks on individual access

* Cleanup and tests

* Address peer review

* Minor greeks fix

---------

Co-authored-by: Martin Molinero <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants