* Fixes Regression Algorithm Testing No Warm Up Requirement
* Removes Volatility Model Warm Up Requirement
If the volatiliy model is not ready, we will use the Brenner and Subrahmanyam (1988) approximation (BS88).
Lean will not set the volatility model if the user doesn't, since the BS88 can handle the null model case.
* Using IV to calculate Greeks, and remove single-step stochastic approximation
* Correct calculation for theta, vega, rho
* Add calculation from Black Calculator and peer review
* Address peer review and added unit test
* Update some tests and correct vega/eho
* Fix Unit Test and Improve Comments
Fixes `IndexOptionCallITMGreeksExpiryRegressionAlgorithm` since `Vega` was really non-zero.
* Fix regression test and add IV calculation
* refactor and bug fixing on peer review
* refactor and bug fixing on peer review
* for rerun test
* add warning on IV estimation not coveraged and edit speed unit test to not exceed 2s per 1000 iteration
* update logging
* update logging and description
* Add default option pricing models and unit tests
* address review
* Added Fed interest rate as risk-free rate with unit tests and set as default for option greeks calculation, added regression algorithms, addressed peer review
* refactor structure of interest rate
* Skip Saturday and Sunday
* regression test fix
* peer review
* Fixes Interest Rate Provider Logic
* Minor tweaks
* Fix start date
* Minor test tweak
* Update interest rates
* Fix unit tests
* Add minor log
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Limit exercise in IB brokerage for limit and cash-settled options
* Add regression algorithm
* Assert that index option is automatically exercised in regression algorithm
* Wolverine LEAN integration
* Wolverine Lean integration - adjusting comments and restrict to Equity only
* fix 0.005$ per share fee
* remove PostOnly from Wolverine as its not needed
* AccountType.Margin and also removed check for USD only currency
* fix Market Wolverine -> USA
* change config
* Minor tweaks
Co-authored-by: Mark Virchenko <mark.virchenko@redduck.io>
* Data monitor (FIRST TEST)
* Calculate data request frequency in data monitor
* Detecting universe data usage in data monitor
* Json data monitor report generation
* Data monitor refactor
* Log and store data monitor report
* Data monitor report tests
* Clean up
* Clean up
* Address changes request
* Address changes request
* Wire data monitor only for backtest, not for live or research
* Minor tweaks
* Some minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Refactor multi IDQH frontier
- Fix for multi IDQH frontier support, specially for IB + QC
- Adding new CompositeTimeProvide. Adding unit tests
* Adjust solution to fix new data available event
* Memory Related Performance improvements
- Make sure we cleanup & dipose of python related objects during pandas
data generation.
- Disable memoizing enumerable use while creating pandas data frames,
since we do not require it
- Reduce unrequired object creations
- Replace concurrentCollections for ordinary locks
* Decimal parsing typo fix
* Add python additional paths for research
* Delegate adding additional paths to pythonpath to PythonInitializer
* Add new way to keep algorithm location at the beginning of pythonpath
* Add switch to disable BeginAllowThreads in PythonInitializer
* Add switch to disable BeginAllowThreads in PythonInitializer
* Address changes request
* Adds Support to Visual Studio Code Dev Container
Adds support and documentation to VSCode Dev Container.
* Address Peer Review
- Adds Dockerfile to install QuantConnect Stubs to enable Python autocomplete.
- Adds settings.json to enable Python autocomplete and LF line ending.
- Removes autobuilder task.
- Fixes Research Laucher
* Launcher and Task Impromevents
Fixes line endings issue
Fixes Launcher to point to right config.json file
Improve readme formatting
* Simplify Jupyter Notebook Deployment
- In some rare cases the FillForwardEnumerator was missing to register
the delisting date of an asset because it was the first data it got.
Adding unit test reproducing issue
* Normalize time in universe behavior
- Normalize Option & Future chain universe behavior regarding their
assets time in universe. They will now respect the universe settings
time in universe value. Adding new regression algorithms asserting the
behavior
* Address reviews & cleanup
- Update QLNet library to latest version from 6/2019 -> 11/2021
- Fix evaluation date which was using settlement date. Adding unit tests
reproducing issue
Use `SetEndDate` to set `EndDate` in `QCAlgorithm` constructor. It will ensure that the `EndDate` is independent of the time the algorithm is executed if `SetEndDate` is not called in `Initilaize`.
Some users don't implement `SetEndDate` to run the algorithm to the latest datapoint, but this is not true if we run the algorithm during the day as the latest datapoint will be 24 hours before the execution time while there is data until the current day midnight.
* Adds Additional Condition to Parse SecurityIdentifier Properties
A string can pass the previous conditions and create an invalid `SecurityIdentifier` (e.g. "Sharpe ration"). If it is invalid, the Market is not supported (e.g. "378").
* Adds Unit Test to Pandas Indexing
* Log Only Once if TryParseProperties Cannot Parse
Use cache to increase speed and remove redundant logging
* docs(VIXDualThrustAlpha): `determine` fixup
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs(VixDualThrustAlpha): `unsubscribe` fixup
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs(BasicTemplateIndia): `Prperties` -> `Properties`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: `Intialization` -> `Initialization`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Get history of a given bar type in python
* Separate GetDataFrame in sub methods to avoid dynamic type resolution
* Housekeeping and unit tests update
* Use all requests in python type method history method
* Address changes request
* Reverted some requested changes
* Handle non-unique multi-index error for Ticks in data frame creation
* Update unit tests
* Update unit tests and add comments
* Add regression algorithm
* Fixes PythonData.EndTime
The `EndTime` property of `PythonData` didn't override the base implementation in `BaseData`. Therefore `EndTime` was always set as `Time`.
Fixes Regression Tests
- The C# version of `Bitcoin` class needs to implement `EndTime`-
- Fixes regression tests
* Minor tweaks and rebase
- Add support for python data types just setting EndTime. Adding unit
tests for PythonData
- Fix for DynamicData EndTime property being fetched. Updating unit
tests
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add algorithm settings to backtest results
* Use algorithm result settings in report generation
* Add unit tests
* Housekeeping
* Changes for live result report
* Housekeeping
* Housekeeping
* Housekeeping
* Add unit tests
* Dynamically set wait time on WS error
- Dynamically adjust wait time on websocket error
* Avoid subscribing multiple times to message events
- Avoid websocket subscription multiple times to message events during
disconnection/connection cycle
* Add Dispose Brokerage WebSocket Subscription Manager
- Add check for live warmup history request, making sure there is at
least room for a single bar between start and end. Avoid sending
start==end when rouded down by the resolution causing some history
providers to fail
- Create SubmitOrderRequest method in algorithm API to reduce
duplication of code and responsability.
- Add order command tests for different failure and success cases
- Fix null reference exception in equity fill model, when there is no
data for an asset.
* docs: `excersizing` -> `exercising`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: fix `exercising for exercising`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: `migth` -> `might`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Support extended market hours in AddFuture()
* Support extended market hours in AddFutureContract()
* Add C# regression algorithm
* Add Python regression algorithm
* Add regression algorithm for future contracts
* Add regression algorithm checking market hour ranges
* Fixed future regression algorithms to use extended market hours
* Fixed future regression algorithms to use extended market hours
* Fixed future regression algorithms to use extended market hours
* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours
* Update future market hours to include extended in market hours database
* Fixed AddFutureOptionContractDataStreamingRegressionAlgorithm to use extended market hours
* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours
* Fixed AddFutureContractWithContinuousRegressionAlgorithm to use extended market hours
* Fixed BasicTemplateContinuousFutureAlgorithm to use extended market hours
* Fixed BasicTemplateFuturesAlgorithm to use extended market hours
* Fix BasicTemplateFuturesDailyAlgorithm to use extended market hours
* Fixed BasicTemplateFuturesFrameworkAlgorithm to use extended market hours
* Fixed BasicTemplateFuturesHistoryAlgorithm to use extended market hours
* Fixed ContinuousBackMonthRawFutureRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureBackMonthRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureHistoryRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureRegressionAlgorithm to use extended market hours
* Fixed DelistedFutureLiquidateRegressionAlgorithm to use extended market hours
* Fixed AutomaticIndicatorWarmupDataTypeRegressionAlgorithm to use extended market hours
* Fixed ConsolidateRegressionAlgorithm to use extended market hours
* Fixed DelistingFutureOptionRegressionAlgorithm to use extended market hours
* Fixed EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm to use extended market hours
* Fixed FutureContractsExtendedMarketHoursRegressionAlgorithm to use extended market hours
* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours
* Fixed FutureMarketOpenConsolidatorRegressionAlgorithm to use extended market hours
* Fixed FutureOptionBuySellCallIntradayRegressionAlgorithm to use extended market hours
* Fixed FutureOptionCallITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionCallITMGreeksExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionCallOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionDailyRegressionAlgorithm to use extended market hours
* Fixed FutureOptionHourlyRegressionAlgorithm to use extended market hours
* Fixed FutureOptionMultipleContractsInDifferentContractMonthsWithSameUnderlyingFutureRegressionAlgorithm to use extended market hours
* Fixed FutureOptionPutITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionPutOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortCallITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortCallOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortPutITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortPutOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm to use extended market hours
* Fixed FuturesExpiredContractRegression to use extended market hours
* Fixed FutureSharingTickerRegressionAlgorithm to use extended market hours
* Fixed HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm to use extended market hours
* Fixed HistoryWithDifferentDataMappingModeRegressionAlgorithm to use extended market hours
* Fixed HistoryWithDifferentDataNormalizationModeRegressionAlgorithm to use extended market hours
* Fixed LimitOrdersAreFilledAfterHoursForFuturesRegressionAlgorithm to use extended market hours
* Fixed OpenInterestFuturesRegressionAlgorithm to use extended market hours
* Fixed RegisterIndicatorRegressionAlgorithm to use extended market hours
* Fixed SetHoldingsFutureRegressionAlgorithm to use extended market hours
* Fixed WarmupFutureRegressionAlgorithm to use extended market hours
* Fixed AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm to use extended market hours
* Fixed AlgorithmHistoryTests to use extended market hours for futures
* Fixed AlgorithmTradingTests to use extended market hours for futures
* Fixed BrokerageSetupHandlerTests to use extended market hours for futures
* Fixed TimeRulesTests to use extended market hours for futures
* Fixed FutureOptionMarginBuyingPowerModelTests to use extended market hours for futures
* Fixed FutureMarginBuyingPowerModelTests to use extended market hours for futures
* Fixed FileSystemDataFeedTests to use extended market hours for futures
* Fixed QuantBookHistoryTests to use extended market hours for futures
* Split BasicTemplateContinuousFutureAlgorithm to have an extended market version
* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours
* Split BasicTemplateFuturesAlgorithm to have an extended market version
* Split BasicTemplateFuturesAlgorithm to have an extended market version
* Split BasicTemplateFuturesFrameworkAlgorithm to have an extended market version
* Split BasicTemplateFuturesHistoryAlgorithm to have an extended market version
* Revert AddFutureContractWithContinuousRegressionAlgorithm
* Revert AddFutureOptionContractDataStreamingRegressionAlgorithm and added data
* Revert AddFutureOptionContractFromFutureChainRegressionAlgorithm
* Revert AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm
* Revert ConsolidateRegressionAlgorithm
* Revert Algorithm.CSharp/ContinuousBackMonthRawFutureRegressionAlgorithm.cs
* Revert ContinuousFutureBackMonthRegressionAlgorithm
* Revert ContinuousFutureHistoryRegressionAlgorithm
* Revert ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm
* Revert ContinuousFutureRegressionAlgorithm
* Revert Algorithm.CSharp/DelistedFutureLiquidateRegressionAlgorithm.cs
* Revert EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm
* Split FutureMarketOpenAndCloseRegressionAlgorithm to have an extended market version
* Split FutureMarketOpenConsolidatorRegressionAlgorithm to have an extended market version
* Revert FutureOptionBuySellCallIntradayRegressionAlgorithm
* Revert FutureOptionCallITMExpiryRegressionAlgorithm
* Revert FutureOptionDailyRegressionAlgorithm
* Revert FutureOptionPutITMExpiryRegressionAlgorithm
* Revert FutureSharingTickerRegressionAlgorithm
* Revert FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm
* Revert FuturesExpiredContractRegression
* Revert HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm
* Revert HistoryWithDifferentDataMappingModeRegressionAlgorithm
* Revert HistoryWithDifferentDataNormalizationModeRegressionAlgorithm
* Revert OpenInterestFuturesRegressionAlgorithm
* Revert RegisterIndicatorRegressionAlgorithm
* Revert SetHoldingsFutureRegressionAlgorithm
* Revert WarmupFutureRegressionAlgorithm
* Revert AutomaticIndicatorWarmupDataTypeRegressionAlgorithm
* Some cleanup
* Address changes request
* Address changes request
* Add more Class III Milk data to fix DelistingFutureOptionDailyRegressionAlgorithm
* refactor(Algorithm): Correct `Intrinino` -> `Intrinio` references
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(Algorithm): Update `evemts` -> `events`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(BubbleAlgorithm): readability improvements
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(OrderTicketDemoAlgorithm): readability improvements
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Fix KeyError when rendering exposure chart in report with some security types
* Update exposure chart colors and add unit tests
* Updated json test file
* handle multiple command files
* remove default useage
* add tests for verify files are read in order
* add comment
* change reuslt file name
* add comment
* liquidate using given symbol values
* remove redundant
* accept ticker, market, security type to build symbol, to be used by CLI commands
* use null leverage
* symbol can be null if ticker, market, security type values are provided
* reject the default value of security type
* cleanup
* cleanup
* check if status is not invalid
* cleanup
* abstraction to remove duplication
* fix name
* address reviews
* use counter
* add test to validate symbol/ticker agrumnets
* remove hardcoded values
* MRP
* Fix bug
* Rename
* Provide virtual `GetPriceRelative` function to override for any variants in future
* SymbolData class within model
* Address peer review
* fix bug
* revise unit test and fix bug in python model
* revise unit test and fix bug in model
* Revise regression statistics
* Revise regression statistics
* Revise regression statistics
* Revise regression statistics
* Revise regression statistics
* Address peer review
* Updated unit tests according to peer review
* Address peer review
* Fix LocalMarketHours.GetMarketClose to get the actual next market close time with extended market hours
The method was not properly considering gaps between regular hours close and extended market open
* Revert some changes to fix errors
* Add performance gh action
- Add new performance github action, running python and csharp benchmark
algorithms
- Adjusting algorithms length so they are not too long or too short
* Checkout master branch benchmark reference
- Fill model will take into account internal configurations, this is
specially relevant for continuous future mapped contracts. Updating
regression algorithms and unit tests.
- Allowing internal configurations to update trade builder information
* Remove suppport for market orders and MOO orders for futures
* Add regression algorithm to assert that limit orders are filled on after hours for futures
* Remove suppport for market orders and MOO orders for futures in DefaultBrokerageModel
* Update regression algorithms stats
* Fixed regression algorithm and clean up
* Add unit tests
* Fix regression algorithms
* Address changes request
* Respect warmup resolution given
- The data feed will respect the warmup resolution given and override
the resolution used by the algorithm when adding a subscription.
Updating regression algorithm to keep previous statistics. Adding new
regression algorithm asserting the desired behavior
* Testing improvements
- Add more unit tests and regresion test
- Add missing data for crypto
- Fix bug with FFed data crossing after the end time of the warmup
request
* Add more Warmup resolution regression algorithms
- Adding more warmup resolution regression algorithms, using
Settings.WarmupResolution and an option selection case
* Add more warmup regression tests
- Adding more warmup regression tests.
- Will no longer skip universe selection subscriptions from warmup
resolution enforcement. Updating regression algorithms data points
* Fix bug with data rounding
- Fix data rounding bug when warmup resolution is set to a different
value than the original configuration. Updating regression algorithms
to assert the expected behavior
* Address reviews
- Revert regression algorithms changes to use Resolution during warmup.
Updating their stats.
- Adding new regression algorithms asserting the behavior warming up
using a timespan and no warmup resolution
- Fix bug where data used to warmup the 'normal' enumerator will make it
through into the warmup time span. Updating tests
* Address reviews
- Add missing comments, explaning warmup algorithms time span
calculations.
- Revert changes in existing `WarmupOptionTimeSpanRegressionAlgorithm`
to reduce diff to minimum
- Adding new warmup unit tests asseting algorithm warmup start time, for
different combinations of bar count, timespan, resolution
* First commit
* Any change
* Add more unit tests
* Nit changes
* Changes requested
* Fix bugs and requested changes
* Requested changes
* More changes
* Check ongoing backtest orders
* Read ongoing backtest does not fail
* nit change
* Fixes for standarized API orders response
- Fixes for a standarized API read orders reponse
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Minor improvements
- Avoid division by zero error in backtesting result handler. Race
condition where the SetAlgorithm implementation would still be running
and we try to loop through the update method
- Improvement nullreference exception message
- Cleanup resolution to timespan extension method
* Address reviews
- Cleanup data config fill forward conditional statement for tick
resolution
* Fix live future and option universe selection daily resolution
- Adding unit test reproducing issue for option and future universe
chain using daily resolution, since was rounding down in UTC TZ the
date might not change and it would re emit the same data point
* Address reviews
- Add more test cases
- Adding locks around RegisteredSecurityDataTypesProvider to allow it to
be used by multiple threads at the same time without exploding
* Minor fixes for mapped futures contract
- Future security was never able to be non tradable
- Skip remove future contracts from live holdings. Adding unit test
reproducing issue and regression algorithm
* Fix removal of continuous futures
- The canonical continuous future would leave behind an OpenInterest
subscription when removed because of a different in the way the
configurations were built, this will now be centralized in a single
method. Adding unit tests reproducing issue
- FillForwarding setting of the continuous future was not being
respected. Updating algorithm reproducing the issue to assert the
behavior failing in master
* Address reviews
- Minor refactors addressing reviews
* Fix result handler live holdings
- Live result handler live holdings will include future assets
associated with continuous futures. Adding unit tests
* Fixed for live holdings price rounding
- Live holdings will use the SPDB minimum price variation for rounding.
Adding unit tests
- Remove thread in charge of sorting and updating the queues weight.
This will be handled by each worker thread instead. Avoid cases where
the work count is so high that the sorting thread in master would not
be able to update all the weights, causing data to be enqueued beyond
the max prefetch limit. Adding benchmark algorithm reproducing issue
- Smaller live warmup history requests, will keep track of the last point
we got from the file based enumerator and start our history enumeration from this point
* Add new Python QCAlgorithm.History() method with all parameters and type
* Add regression algorithms
* Using all parameters in History()
* Use private methods to reuse History() code
* Use private methods to reuse History() code
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Asserting history count
- Similar to the OptionChainUniverse, the future chain universe should
also add it's contracts using the raw data normalization mode. Adding
regression algorithm reproducing issue
- Replace logic to convert TimeSpan based PeriodConsolidator into count
based, by an override of the 'potentialStartTime' in the case we could
be falling into a look ahead consolidated bar end time
- Adding OpenInterestAnnual where the contract maps when any of the back month
contracts of the next year have a higher volume that the current front month.
- Add support for backwards compatible data mapping mode additions
- Updating ES map and factor files. Adding a little daily data too
- During warmup period the algorithms initial time might not be a rounded date
value, so it's important to take into account hours/minutes. Adding
regression algorithm reproducing and asserting issue
* Period timespan consolidation improvements
- If user is trying to consolidate a period providing data of a bigger
period we will now throw an exception. Adding tests
- If both consolidated and given data share the same period, gently
adjust the consolidator into a data count of 1. Adding tests
- Fixing bug in QuoteBarConsolidator period double accounting. Adding unit tests
* Add Period and Count regression algorithm
* Fix QuoteBar NonTimeBased Consolidator Period
- Fix QuoteBar non timebased consolidator period, that was accounting
twice for the initial bar period. Updating unit tests
* Improve QuoteBarConsolidator assertion
- Assert quote bar consolidated time and endtime in unit tests
- Add error log at OptionSymbol.IsOptionContractExpired in the case we
detect an unexpected expiration time. That could be related to
Symbol.ID.Date being incorrect
* Fix option price and greeks always beign zero on the expiration date
* Fix option price and greeks always beign zero on the expiration date
* Updated documentation
* Style and performace changes
* Address changes request
* Fix QL.Settings.includeReferenceDateEvents not being set on every thread
* Updated code documentation
* Add indicator for allowed option styles to QL option price models
* Add and update option price model tests
* Update option price models methods sumary to indicate allowed option styles
* Add regression algorithms for option price models for different option styles
* Update OptionPriceModel regression algorithms to use Lean local data
* Add Python regression algorithms for option price models for different option styles
* Update OptionPriceModel regression algorithms to assert that greeks are valid
* Address changes request
* Address changes request
* Update OptionPriceModel regression algorithms to check both call and put contracts
* Update OptionPriceModel regression algorithms to use correct test data
* Update OptionPriceModel regression algorithms to throw in OnData
* Address changes request
* Update OptionPriceModel regression algorithms to assert greeks are not all zero
* Live T-1 selection warmup
- Only use BaseDataCollection class
- Reuse collection enumerator
- Remove FuturesChainUniverseSubscriptionEnumeratorFactory
- Universe selection will use Cache providers
- Add null reference check
- Add more tests
- Fixes for warmup fill forwarding. Add more tests
* Address reviews. Add missing comments
* Refactor chain providers
- ChainProviders will use quote, open interest and trade information,
which ever is first. Updating regression algorithm changing contract
being added.
- Add base BacktestingChainProvider for shared logic, reducing
duplication.
- ChainProviders will now use the given IDataCacheProvider and
IMapFileProvider
- If Providers are called for a date the exchange is closed they will
search for the previous available date where the exchange is open.
Adding unit tests.
* Address reviews. Logging improvements
* use research-object-store-name key from config while creating object store for research notebooks
* initial removal
* remove reference
* move test to private repo containing downloaders
Since there is `UpdateStopPrice`, `UpdateLimitPrice`, `UpdateQuantity` and `UpdateTag` method to make it easier to update the `OrderTicket`, we include `UpdateTriggerPrice` to cover the `LimitIfTouched` case.
Also, updated the docs or the arguments for these methods.
Updates LimitIfTouchedRegressionAlgorithm:
- Adds `UpdateTriggerPrice` call that rounds down the `TriggerPrice`.
* use research-object-store-name key from config while creating object store for research notebooks
* remove bitfinex files
* remove redundant ref
* move test to private repo
* Warmup minor fixes
- Result handler fix
- Centralize and normalize status update during warmup
* Address reviews
* Minor adjustments
* Add console message during warmup period
* Fix warmup percentage update
* Overload PandasConverter.GetIndicatorDataFrame to accept a Python dict
* Shared implementation code for PandasConverter.GetIndicatorDataFrame overloads
* Added documentation for private shared methods used by PandasConverter.GetIndicatorDataFrame
* Added unit tests for PandasConverter.GetIndicatorDataFrame
* Added unit tests for PandasConverter.GetIndicatorDataFrame Dictionary overload
* Address change requests
* Address change requests
* Reproduce TrailingStopRiskManagementModel bug with test case
* Fix TrailingStopRiskManagementModel to make it relative to max price
* Adapted old TrailingStopRiskManagementModel test to new implementation
* Fix TrailingStopRiskManagementModel Python version
* Fixed TrailingStopRiskFrameworkAlgorithm regression tests data
* Handling both long and short positions in TrailingStopRiskManagementModel
* Traking holdings value instead of unrealized profit in TrailingStopRiskManagementModel
* Checking for position side change in TrailingStopRiskManagementModel
* Handling immediate liquidation in TrailingStopRiskManagementModel
* First commit
* Add unit test
* Change implementation
* Add unit tests
* Nit change
* nit change
* Remove unnecessary methods
* Add more unit tests
* Revert "Add more unit tests"
This reverts commit 1ba2ab7454.
* Add more unit tests
* Add more unit tests
* Improve Implementation
* Change unit tests
* Remove unit tests
Remove unit tests from AlgorithmRegisterIndicatorTests.cs
* Coarse fine flag improvement
- Coarse will check global data folder for fine existance if it's
different than the given path to check
* Update readme.md
- Minor fix for FuncPeriodSpecification so it always uses the same
DateTime to assert the given function. Improve documentation and
exception message being thrown
* Add missing stablecoins
* Fix bugs
* Remove MIMUSD stablecoin pair
That pair is already in the SPDB
* Add more stablecoins pairs
- Add SUSD and IDRT
* Fix bug
* Requested changes
The algorithm posted in the GH was failing because when finding a conversion between currencies BTC to USD, it wasn't taking into account BTCUSDT and that USD = USDT because is a stablecoin
* Solve bugs
* Nit changes and more tests
* Requested changes
* Nit changes
* Requested changes
* Nit change
* Allow DataWriter to merge existing ticks if desired
- Optionally allow lean data writer to merge new ticks with existing
- Adjust ZipStreamWriter
- Fix bug in compression lib that was leaving files open. Reproduced by
added tests.
* Address reviews
- Define new WritePolicy to configure the behavior of the LeanDataWriter
regarding how it should handle writing to a file: merge, overwrite,
append
* make public safe to expose test properties
* extract ib files
* remove dependencies
* refactor to use ib reference as string
* remove un-used dependencies
* remove redundant refrences
* Added checking if algorithm is stopped in initialize method
* Quit on initialize adjusment
- Minor adjustments to solution. Adding more regression algorithms
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Update Tick.cs Time parsing
* Adjust equity tick millisecond read
* Fix tick sub millisecond precision
- DateTime.AddMilliseconds will rount to the nearest integer. So instead
use AddTicks
* Avoid sporadic test failure
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* set price currency for open orders from brokerage setup handler, remove setters from brokerage
* order already has price currency being set by brokerage transcation handler
* remove as overwritten by BrokerageSetupHandler.GetOpenOrders
* refactor because can't use internal setters in private repos
* refactor, use ctor to set attributes
* remove redundant broker dependency
(cherry picked from commit 57b071eb174ca9fe857021ebffa7a546834e86f7)
* check limit price for limit order instead for checking price
* add price agrument for MarketOrder
* remove, not required anymore
* AddUnrequestedSecurity should not return null
* assert string in not null cases
- Fully reset the security cache when it's finally removed from the algorithm.
Adding regression algorithm reproducing issue
- Updating regression algorithms which would trade based on data
previously available
* Warm up EmaCrossAlphaModel
Warm Up EmaCrossAlphaModel indicators
* Fix regression test bug
When using the default `EmaCrossAlphaModel()` the period of both indicators to be ready is bigger than the difference between the start date and the end date of the algorithm. Then, as the algorithm didn't warm up the data both indicators of EmaCrossAlpha never were ready, but now as the model warms up the data both indicators are ready so we get different statistics
* Requested change
* Fix unit tests
As there wasn't items in `AddedSecurities`, when trying to remove the items in ´RemovedSecurities´ there was nothing to remove because there was never a security in `_symbolDataBySymbol`. That's why, in order to test, the behavior of `EmaCrossAlphaModel` when removing a security we need to first add one to then remove it.
* Requested changes in Python
- Requested changes in Python
- Nit changes
* Nit change
* Requested Changes
* Add RemoveConsolidators() method in Python version
* Add requested History overload
* Add more regression tests
* Revert "Add more regression tests"
This reverts commit 71b279e917.
* Add more regression tests
* add data count properties
* 'add history count property
* assert data counts
* update missing override
* consider override/virtual cases
* implement data count
* add message handler for regression tests
* use regression test message handler
* set algorithm manager for regression test message handler
* update data count
* check if stats are present, check if algo manager is not null
* update
* add c# algo
* make same as c# algo
* use new line
* logic shifted to RegressionTestMessageHandler
* cleanup
* auto cleanup
* skip non deterministic data count
* change data count
* use inheritance
* improve stats
* update couht
* add sma indicator to c# and customSMA to python
* call base method before executing further
* skip test
* revert to original
* add duplicate sma
* skip regression test
* Make continuous futures untradable
- To match live trading behavior. Adjust continuous futures securities
to be untradable
* Set ContinuousFuture as non tradable
- Set continuous futures as non tradable. Update regression algorithms
- Fixes for symbol capacity calculation
`WarmUpIndicator` for Python indicators doesn't return the indicator anymore after #6027. So all overloads should return `void`.
Fixes `SmaCrossUniverseSelectionAlgorithm` [C# and Py].
* Added NYSELIFFE exchange support to Interactive Brokers brokerage.
* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].
* Added NYSELIFFE exchange support to Interactive Brokers brokerage.
* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].
* Added NYSELIFFE exchange support to Interactive Brokers brokerage.
* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].
* - Added NYSELIFFE futures to Futures.cs
- Added method GetGoodFriday() to calculate Good Friday to FuturesExpiryUtilityFunctions.cs.
- Added unit tests for GetGoodFriday().
- Added NYSELIFFE futures expiries to FuturesExpiryFunctions.cs which uses GetGoodFriday() instead of hardcoding the date of Good Friday in the market-hours-database.json.
- Added unit tests for NYSELIFFE futures expiry.
- Added NYSELIFFE to Exchange.cs
- Removed Equity-nyseliffe-[*] and Index-nyseliffe-[*] from market-hours-database.json
- Updated Option-nyseliffe-[*] to FutureOption-nyseliffe-[*] on market-hours-database.json
- Added the supported NYSELIFFE futureoptions to symbol-properties-database.csv.
* Removed blanks after India tuple
* Added the dates for 2022 and 2023 to the market-hours-database.json.
Removed "FutureOption-nyseliffe-[]" from market-hours-database.json.
Added to FuturesOptionsSymbolMappings.cs:
{ "YG", "OYG" },
{ "ZG", "OZG" },
{ "ZI", "OZI" }
* Minor tweaks for nyseliffe FOPs
Co-authored-by: Chris Coetzee <chris@polidata.ai>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Deprecate breakpoints
Drop Watchlist
Remove breakpoints from lean
Prep install of NetCoreDbg
Cleanup
* Install NetCoreDbg
* Update Research Images with Newer Jupyter
* Binance.US base changes
* biniance exchange info update
* Binance US uses same fee rates as main Binance
* binance us brokerage model tests
* use base implementation of GetBuyingPowerModel method
* add gdax exchange info downloader
* add downloader method to program]
* fetch currency description
* change definition to include headers
* use extension method to make request
* remove log from test
* replace WebRequest
* cleanup
* use relevant name
* implement IExchangeInfoDownloader for bitfinex, initial commit
* add default values
* use default market value
* use correct attribute for lotsize
* don't skip missing values
* handle multiple downloaders
* add gdax and bitfinex exchange downloader
* follow LEAN data directory structure
* update SPDB
* order tickers
* order tickers
* add exchange info downloader test template
* delete files
* update SPDB
* use currency mapping
* update bitfinex symbols
* update currency mapping
* sort result after old currency symbols are used
* use market of the respective brokerage
* no more unknown symbol
* change minimum order size value
* direct conversion possible
* update bitfinex symbols
* change user-agent
* add test for indirect conversion
* update stats
* remove un-used
* use utc time for slice sync
* use utc time
* refactor
* add regression test
* use utc time
* use utc time
* make utctime required parameter
* add utcTime in slice creation
* assert warm up complete
* check if algorithm is still warmingup
* use exchange tz
`CashBuyingPowerModel`, which reflected on `AlphaStreamBrokerageModel`, a margin-only brokerage.
It also didn't consider the account type, so `InteractiveBrokersBrokerageModel` was using the margin model even if `AccountType.Cash` was selected.
Removes `GetBuyingPowerModel` method from other Brokerage Models when their cases are covered by `DefaultBrokerageModel`
Fixes some typoes in `TradierBrokerageModel`
Fixes unit and regression tests. For the regression tests, we have explicitly set the brokerage model.
- The BrokerTransactionHandler will also adjust virtual positions for
margin accounts when fees are in base currency and the asset is a
crypto or forex pair. Adding new regression tests reproducing issue
* extract binance brokerage
* remove binance tools from main toolbox app
* modify OrderTestParameters constructor
* fix BinanceBrokerage ref config.json
* update symbol properties
* move BinanceFeeModel tests to the right place
* initial commit
* get history from data providers
* merge history from history providers
* merge slice
* Add merge function in slice
* Update test
* merge aux data
* add data points in _data
* append new data points in original list
* Add test suite for HistoryProviderManager
* reduce complexity
* setup once
* add fake history provider
* doesn't count aux data
* add tests for options
* style changes
* add custom data
* merge rawDataList
* use array of history providers
* optimize
* fix formatting error
* add comment
* add tests
* simplify
* use list
* use abstraction
* split tests
* add test
* use abstraction to create generic enumertor class
* refactor
* accept list of type T
* sync history slices
* use SubscriptionDataReaderHistoryProvider for live
* rename test file
* return empty
* add tests
* cleanup
* address reviews
* optimize
* Fix method definition
* use initial time for basedata
* refactor
* re-use collection
* inherit HistoryProviderBase
* always return HistoryProviderManager
* add tests
* update rawDataList
* reset composer
* consider null elements
* add tests for binary search method
* Follow lean coding style
* revert
* remove binary search method
* convert to field
* Update MHDB and SPDB
Update MHDB and SPDB with micro futures
* Fix bugs
* Add more FutureExpiryFunctions
* Requested changes
* Nit changes
* Nit changes
- Fix some future values in SPDB
- Test LastFriday() method
- Apparently Micro CHF/USD Futures(MSF) refers to the micro of Swiss Franc Futures (6S). The same happens with Micro JPY/USD (MJY) Futures and Japanese Yen Futures (6J)
* Fix bugs
* Micro futures MultipleFactor
* Fix some futures MultipleFactor
* Adding unit tests reproducing issue.
* Fix a couple of minor bugs
- IsMarketOpen will work correctly when used with daily and hourly
resolution.
- slice.Get will work correctly with python custom data
- ExtendedDictionary will be able to dinamically access methods,
required for python and private C# data types
* Refactor solution. Add more tests
* Remove unrequired import statement
* WIP
* Fix fees in base currency not being subtracted
- Fix fees in base currency not being subtracted from the quote currency
for crypto cash accounts. Updating regression tests to assert
portfolio, cashbook state and holdings state.
* Fix unit test race condition
* Use GC server mode for tests
* Fix daily auxiliary data points emission time
- Due to fillforwarding, in some cases with daily resolution symbol
change events (generically any auxiliary data) would arrive late.
Updating regression test to reproduce the issue. Adding unit test
- Some refactoring and logging improvements
* Address reviews
* Remove old xml docs param
* change BinanceBrokerageModel
* supply spot vs margin endpoint as parameter
* wip
* send margin order for margin account
* "NEW" means that the order has been accepted by the Binance engine.
* fix cash balance
* use JsonConverters for account parsing
* unit tests
* fixup
* lazy connect
* fix connection
* allow api client to be null if DQH only
* make method private
* more unit tests
* fix Dispose
* fix tests
* fix IsConnected condition
* run test as additional
* add some comments
* improve unit tests
* improve null checks
* tidy up the code
* Refactor `GetFilePath()`
Add also useful methods to use with this one
* Nit changes
* Requested changes
* Requested changes
* Restore SaveString()
* Nit changes
* Address self review
* Test improvements
* Adjust example KerasNeuralNetworkAlgorithm
* Minor tweak for KerasNeuralNetworkAlgorithm.py
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Binance fees deducted from fill quantity accordengly
- For Binance cash accounts while buying, if fees are from the base
currency of leans virtual position, we need to deduct the fee from the
fill quantity, else we can end with a position bigger that it actually
is and not be able to liquidate
* Refactor solution
- Refactor solution into a more generic approach solving fees in base
currency at the BrokerageTransactionHandler level, covering all
brokerages that require it. Adding regression algorithm reproducing
issue.
- Update Bitfinex and Binance fee models to correctly reflact reality
* Log fill quantity adjusment once
* Add support for live price scaling
- Add support for live trading price scaling for continuous futures.
Adding unit tests
* Move live price scale application. Updating unit tests
- Live trading application of price scaling will happen before fill
forwarding and updating securities real time price. Updating unit
tests to reproduce issue
- GetLastKnownPrices will no longer guess which resolution to use but
rely on other methods implementation/
- Updating basic template future algorithms to warmup contracts and
assert it
- Minor improvements for FunSecurityInitializer and FuncSecuritySeeder
* Always reuse aggregator instance if any
- When fetching a IDataAggregator instance from the composer, do not
enfore type name on existing instances
* Fix unit tests
* add data
(cherry picked from commit 814011d89e5316d150f88ffca5f48d8d5f0ea7d9)
* update market hours for index
(cherry picked from commit edac40732c120eb84d27de00594b59eebb4983f5)
* add index algorithms
(cherry picked from commit b22d27b4fa98172c435f7c26de4a3a297c49a6b7)
* update statistics
* add cash
* Add india market
* add leverage for index
* can subscribe to index
* update format
* fix wrong cash
* fix ticker names
* update data
* update ticker and stats
* update docs
* specify enpoint url of ftx api (can be FTX pr FTX.US)
* more metadata for ftxus
* tidy up code
* tests
* more tests
* fix tests
* fix us fee rates
* add account tier
* update symbol props
* typo
* typo-2
* update symbol properties
* use FTXUS fee model
* minor tweaks
* replace to local functions as they are more performant
* fix random generator upper bound
Next() includes minValue, but not maxValue, so we increment it +1
* introduce abstract layers
* refactoring
* fix tets
* adapt tests
* fixup
* implement blackschole price model for options
* use risk free rate
* use ql price model
* wip
* change interface
* fix
* tidy up the code
* wip
* iterate groupped symbols
* wip
* wip
* fix
* allow symbol of different types
* improve settings
* wip
* iterate full range
* fix issue with negative option
* fix
* fixup
* use StandardDeviationOfReturnsVolatilityModel
* re-use existing tick types per security type
* parametrize underlying security type
* use default option style
* dynamic option price model
* fix enumeration
* test
* fix unit tests
* refactor code
* remove unused file
* minor tweaks and refactoring
* rename symbol generator class
* fix interface
* add comments
* more comments and unit tests
* more tests
* add disclaimer
* more tests
* more comments and tests
* split tests into different files
* tidy up the code
* tidy up the code; more tests
* refactor TickGenerator => use security price directly on each iteration
* remove dupe; reuse main constructor
* use SecurityManager, refactor code
* bugfix: save ticks in history array
* check volatility warm up & tests
* more unit tests
* describe volatility period span in settings
* rename command line option
* Minor adjusments. Address review
- Use Lean log handler instead of writting directly to console
- Rename BlackShcolesPriceGenerator to generically OptionPriceModelPriceGenerator
- Minor format clean up & standarization
- Add support for specifying the option chain size
* Rename TickGenerator private fields
* Fix unit tests
* fix tests class name
* Support tickers being specified
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Zip entries will be sourced from cache provider
- Option underlying will use SubscriptionDataSource to fetch it's data.
Fixing bug where it would let through an old data point, or miss
sending data through.
* Add india market data
* use local data for algo
* Add India market regression algo
* update india market data
* Update readme
* add python algo for BasicTemplateIndiaAlgorithm
* Add India data regression python algo
* data india data files
* update tickers
* Fix algo template
* remove data
* fix stats
* update stats
* remove unused data
- Live subscription enumerator will clone the underlying data set when
live mapping is being done. To avoid issues where IDQH implementations
could reuse a data point with same configurations. Adjusting unit test
to reproduce issue
* Fix typo
* Fix TryGetValue to return false if key is not found
* Revert "Fix TryGetValue to return false if key is not found"
This reverts commit b85b7b579a.
* Update documentation for TryGetValue
* Move Optimizer-related DTOs and JSON converters into Common/Optimizer
* Add REST methods for Optimization
* Move OptimizationStatus into Common
* Change optimizationId parameter type to string
* Update Optimization and add lightweight optimization object
* Rename lightweight optimization to BaseOptimization and remove unneccessary properties
* Remove snapshotId from Optimization, add ParameterSet to Backtest
* Add missing IApi.cs method signatures
* Move ParameterSet into Common
* Replace Backtest with OptimizationBacktest
* Update UpdateOptimization to not include null or empty name and layout params in the request
* Change Objective targetTemplate regex pattern from ['(.+)'] to (.+) to prevent escaping target strings without whitespace
* Return Estimate object when calling EstimateOptimization
* Use DefaultNamingStrategy when serializing constraint operators
* Revert "Change Objective targetTemplate regex pattern from ['(.+)'] to (.+) to prevent escaping target strings without whitespace"
This reverts commit fbe7de0fd7.
* Update Api method signatures
* Add unit tests
* Fix XML comment referring to the old class name
* Fix XML summary for OptimizationResponseWrapper
* Address review feedback
- Remove unused testOrganizationId
- Change NodeType from string to NodeType enum
- Clarify unit types for Estimate time and balance
- Simplify JsonConverter classes
* Add accessors to Common/Api classes
* Define performance metrics names in PerformanceMetrics class
* Remove unnecessary branching logic from GetSeriesValues method
* Add crefs and examples to XML comments in the Api class
* Revert "Change NodeType from string to NodeType enum"
* Remove layout param from UpdateOptimization method
* Backtest property ParameterSet should be of type ParameterSet
* Add asserts for deserialization in OptimizationBacktestJsonConverterTests
* Replace the three target-related properties with Criterion
* Add serialization and deserialization tests for Optimization
* Remove Optimization Serialization test
* Add EstimateDeserialization test
* Add asserts for integration tests
* Address self review
* Revert test case
* Update Nodes.cs
* Update Nodes.cs
* Set Aborted status when Optimization fails to start
* Add ParameterSetJsonConverter and ParameterSetJsonConverterTests
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Do not send internal SecurityChanges to Algorithm
- Will not send internal security changes to the Algorithm by default.
Following custom security changes filter pattern. Updating regression
algorithms to assert behavior.
- The universe member will know wether it was added with internal
configurations or not
* Address reviews use a separate collection for internals
* Refactor solution. Adding security changes constructor class
* GH-6115: adjust dates
* fix
* use another approach
* Add lean data writter multiple days data unit test
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Handle security added and removed in the same loop
- Correctly handle adding and removing a security in the same loop.
Adding regression test
* Adding comments
* Align universe security and configuration TZ
- Make sure universe security and configuration tz are aligned always.
Adding unit test reproducing issue
* Address review UserDefinedUniverse will use UTC TZ
* CompositeDataQueueHandler handles duplicate configs
- CompositeDataQueueHandler will handle duplicate subscriptions sharing
the same config. Adding unit tests
* Refactor bug solution after review
- LiveSubscriptionEnumerator will not long perform any symbol mapping
but will just trigger a new subscription call when remapped.
- CompositeDHQ will handling symbol mapping and keep track of these
mapped configs to trigger unsubscribe accordengly. Adding unit tests
* Adjust solution after review
- To avoid breaking uniqueness of SubscriptionDataConfig will be adding
a private `mapped` property that will change equality of mapped
configs to avoid them clashing. Adding unit tests
* Rename IDQH subscription extension methods
* Minor performance improvements
* CompositeDataQueueHandler is UniverseProvider
- CompositeDataQueueHandler implements the UniverseProvider interface.
Adding unit tests
* Indicator
* Fixed logic error
* Removed old files, added KER into KAMA
* Removed old comments
* Added requested changes
* Minor clean up
* Refactors KaufmanEfficiencyRatio and KaufmanAdaptiveMovingAverage
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
* Updates KeltnerChannels to Update MiddleBand with EndTime
* Adds Unit Test
This unit test shows that the timestamp of the MiddleBand and the indication are aligned as expected.
* initial commit
* Follow IDQH implementation
* Expect a list of data handlers from LiveNodePacket
* Return null if can not subscribe
* refctor to add check for subscription
* initialze null
* Add tests
* Check subscribe retuns null/not-null
* cleanup
* Read all required IDQH credentails to job
* Use CDQH to handle all IDQH instances
* constructor abstraction to call from setjob
* use flag
* remove redundant because derived will call initialize on it
* abstract and initialize from setjob
* handle null enumerators
* get creds from data handlers
* handle single data handler value from data-queue-handler
* Fix to support a json array
* Fix missed constructor call
* change access modifier to access from Tests files
* Add test to get brokerageFactory from dataQueueHandler
* Fix init flag to handle all conditions
* Add docs
* initialize from setjob
* Check if websocket open before using
* change defination of initialzie to include tradier
* clean up
* change defination
* fix wrong api key name
* return empty enumerator
* check websocket open before sending request
* check connection before subscribing
* fix to include more cases
* check websocket open before sending request
* Minor refactoring
* reafctor and use IsConnected
* remove unused
* clean up
* Fix test cases
* reverse change
* include config changes
* connect to websocket from setjob
* check websocket connection from setjob
* clean up
* include condition for IDQH that are not brokerage
* Address review
* Add market check condition before subscribe
* Remove deprecated
* Minor fix for deserializing data queue handler
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Fix for adding continuous future and manually adding contract at the
same time. Continuous future was using the user defined universe
symbol and caused clashes. Adding regression test reproducing the issue.
- Fix for IB option position update race condition, where the BTH would
incorrectly assume it should trigger an early option assignment. Adding unit tests
* Give `code()` more time per iteration
* Another proposed solution
* Revert "Another proposed solution"
This reverts commit 29bf5a7554.
* Add an AutoResetEvent field in `TimeConsumer()`
* Avoid change TimeMonitor implementation
* Remove changes in `TimeConsumer.cs`
* Requested changes
* Avoid re-implementation
* Continuous Futures Refactor. Live Mappings
- Adding support for live mappings. LiveTradingDataFeed will handle any
symbol mapping at the configuration layer and resubscribe through the
IDQH
- Refactoring continuous futures adding ContinuousFutureUniverse that
will select the currently mapped security
* Minor fixes
- Remove addition of configurations in UniverseSelection step, leave
resposability for universe.
- LiveTradingDF future unit test will only assert slice data for non
internal feeds.
- ContinuousContractUniverse will respect internal option interest
subscription
* Address review
* Properly handle pending CancelOrderRequests
To avoid unneccessary "Insufficient buying power to complete order" errors showing up as Invalid orders we need to skip further processing of pending CancelOrderRequests in the BacktestingBrokerage and instead let them be properly removed during the next transaction handler run.
* Update BacktestingBrokerage.cs
* Activate Index and IndexOptions low res
* Add Index and IndexOption low res data
* Add IndexOption regressions
* Fix bug with IndexOption QuoteBar data not being scaled
* nit - update comment for _scaleFactor
* Update data and regressions to reflect fix in data
* Tweak regressions
* Fixes for FutureOptions support in LeanData
* Add CreateCanonicalOption() utility function for Symbol.cs
* Add aggregated Futures/FuturesOptions data to Lean
* Add FutureOptions regressions for daily/hourly data
* Allow Futures to be added with low resolution
* Add Future regressions using hour/daily data
* Nit - Python Class names
* Add reviews
* Add alias into CreateCanonicalOption
* Add tick type to data downloader get (price data) call in order for api's with rate limits to handle not making an api call at all.
* Added DTO object to pass into data downloader get
* Added missing header for new file
* Address review. Minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix for python enumerable data history request
- Fix for python enumerable type data history request. Adding unit test.
* Add methods for adding data points into a baseDataCollection
* Move processing of delistings to Brokerage
* Deal with case that exchange is not open on OptionSymbol.ID.Date
* Refactor solution to use DelistingNotification event
* Adjust some regression expected liquidation time
* Mark some todos on deprecated functions
* Update expected liqudation time for Py regressions
* Update regressions that have been validated
* Use HandlePositionAssigned for assignment orders
* Update regressions
* Update some missed unit tests; remove one that is already covered by regression
* Cleanup deprecated backend functions
* nit - small cleanup adjustment
* Post rebase fix
* Address review
* Minor tweak to py regression
* Super Trend Indicator #4653
* Updated test data
Previous test data was wrong
* Reduced if statement
* Updated tests
Replacing spy test data with dwac test data from trading view.
* Minor comment update
* Minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* First BetaIndicator prototype
- In construction
* Fist BetaIndicator version and unit tests
* More unit tests and regression test
* Nit change
* Requested changes
* Nit changes
* Requested changes
* Adjust beta formula slightly and nit changes
* Nit change
- Add new CustomWeight PCM for alpha streams
- Add new AlphaStreams AlphaModule that will handle security additions
and removals, removing this logic from AlphaStreamsBasicTemplateAlgo
* Remove user plan enum
- Get maximum order and runtime from job controls
- Remove user plan enum definitions
* Add log for BacktestingSetupHandler maximums
* Create generic writing for LeanDataWriter, + notes on todos
* Make Options Daily/Hourly data store by year
* Refactor Generic Write
* Permit hour and daily resolutions for options
* Refactor writer to merge when needed with other files
* Cleanup redundancies, run write tasks in parallel
* Make needed classes/vars available
* Update tests to reflect new naming convention for daily hourly options data
* Add Byte[] overloads for ZipData functions in compression
* Implemented Store() for ZipDataCacheProvider
* Have LeanDataWriter use a DataCacheProvider
* ZipDataCacheProvider cleanup
* ZipDataCacheProvider tweaks, doesn't support storing non-zips
* Test adjustments
* Update LeanDataWriter to use Write instead of SaveDailyHourly/SaveMinuteSecond
* Implement tests to verify DownloadAndSave behavior
* Nit cleanup on DownloadAndSave tests
* Fix for options daily/hourly underlying equity subscription read
* Add daily/hourly options data and regressions
* Add missing open interest for hourly
* Fix writing of OpenInterest Daily/Hourly data
* Update data
* Fix Date typo in regression
* Use daily algorithm to test delisting
* Revisions part 1
* Expand test for DataCacheProviders; refactor DiskDataCacheProvider
* nit - test adjustments
* ZipDataCacheProvider test setup refactor
* Adjust multithreaded read/write test; fixes for ZipDataCacheProvider
* Move DiskDataCacheProvider to its own file and add write test
* Remove _appendToZips; always overwrite entry or create zip
* Add mapping regression for daily options
* nit - add license to regression
* Fix Tick write case where more than one data point for a DateTime
* Fix data issue
* Address review
* Tweaks for tests
* Stop Store() early if no entry name is given
* Warm up MACD indicators
- When a security is added in the MACD alpha model, it's warm up at once
* Add unit tests
* Nit change
* Code style and nit changes
* Implement IIndicatorWarmUpPeriodProvider
- Implement IIndicatorWarmUpPeriodProvider in PythonIndicator.cs
- Make a unit test to check whether the WarmUpPeriod is working as expected
- Make a regression test to check the new feature at a system level
* Nit change
* Change Period parameter for WarmUpPeriod parameter
- Change regression test to check if the new parameter keep backwards compatibility with indicators that do not set WarmUpPeriod
* Documentation change
* Fix tests bugs
- In CommonIndicatorTests.cs before finish the test it checks the period.value with the number of samples but for default the period.value was set to -1
* Change names
* Change WarmUp and RegisterIndicator methods
- Lean WarmUp indicator skip custom python indicators that don't define WarmUpPeriod parameter
* Call WarmUpIndicator manually
- Add a new "bridge" method called WarmUpIndicator in QCAlgorithm.Python.cs to set up everything to call WarmUpIndicator in QCAlgorithm.Indicators.cs
- Change the regression algorithm to warm up the indicators manually
* Remove unnecessary code and add more tests
* Nit change
* Revert "Nit change"
This reverts commit da411f59c9.
* Fix bugs
* Try fix bugs
* Add C# regression test
- More nit changes
- Fix bugs
* Requested changes
* Remove unnecessary code
* Requested changes
* Nit changes
- Add new Python class to check a custom indicator, which doesn't inherits from PythonIndicator, warms up properly
* Reduce redundant code
* Fix bug and add more unit and regression tests
* - Add more unit tests
* Nit change
* Test cleanup
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Remove unnecessary attribute `_period` in MarketProfile
- Set the value of WarmUpPeriod to the period in MarketProfile constructor
- Fix unit test to check VolumeProfile and TimeProfile indicators are being warmed up properly and its WarmUpPeriod parameter isn't zero
- If available, CoinApi converter will read previous date files to
rescue initial midnight ticks for the processing date. Also make sure
to drop ticks from tomorrow
* Implement IIndicatorWarmUpPeriodProvider
- Implement IIndicatorWarmUpPeriodProvider in PythonIndicator.cs
- Make a unit test to check whether the WarmUpPeriod is working as expected
- Make a regression test to check the new feature at a system level
* Nit change
* Change Period parameter for WarmUpPeriod parameter
- Change regression test to check if the new parameter keep backwards compatibility with indicators that do not set WarmUpPeriod
* Documentation change
* Fix tests bugs
- In CommonIndicatorTests.cs before finish the test it checks the period.value with the number of samples but for default the period.value was set to -1
* Change names
* Change WarmUp and RegisterIndicator methods
- Lean WarmUp indicator skip custom python indicators that don't define WarmUpPeriod parameter
* Call WarmUpIndicator manually
- Add a new "bridge" method called WarmUpIndicator in QCAlgorithm.Python.cs to set up everything to call WarmUpIndicator in QCAlgorithm.Indicators.cs
- Change the regression algorithm to warm up the indicators manually
* Remove unnecessary code and add more tests
* Nit change
* Revert "Nit change"
This reverts commit da411f59c9.
* Fix bugs
* Try fix bugs
* Add C# regression test
- More nit changes
- Fix bugs
* Requested changes
* Remove unnecessary code
* Requested changes
* Nit changes
- Add new Python class to check a custom indicator, which doesn't inherits from PythonIndicator, warms up properly
* Reduce redundant code
* Refactor error message for reproducability
* WIP Refactor GetMaximumOrderQuantityForTargetBuyingPower
* Additional tweaks and tests
* Address shorted margin case
* Reinstate tests with new modified function
* Update regression
* Some cleanup
* Expand test set
* Refactor solution
* Address review
* Final adjustments and cleanup
* Expand error message for GetAmountToOrder for safety and reproducibility
* Address reviews
* Add base class for pivotpointshighlow
* abstracts do not work well with python switch to virtuals and use pivotpointhighlow as base implementation
* added missing documentatino header
* moved reset back to original location
- Implement IIndicatorWarmUpPeriodProvider interface in both indicators
- Add test to check the new functionality is working as expected. To test WindowIndicator IIndicatorWarmUpPeriodProvider there was used WindowIdentity indicator tests because that indicator only inherits from it
* Update SPDB with price magnifier parameter
- Add a new column for new parameter in symbol-properties-database.csv
- Change InteractiveBrokersBrokerage.cs implementation of GetContractPriceMagnifier() method in order to get the parameter directly from the SPDB
- Add test in SymbolPropertiesDatabaseTests.cs to check if the new parameter is loading properly
* Change GetContractPriceMagnifier() method name and implementation
- Change NormalizePriceToLean() and NormalizePriceToBrokerage() methods
* Nit change in GetSymbolPriceMagnifier() method
* Remove GetSymbolPriceMagnifier() method
- Remove redundant and unnecessary parameters in NormalizePriceToLean() and NormalizePriceToBrokerage()
* Requested changes and SPDB update
- Update SPDB with minimum order size parameter for futures in cents. The minimum order size parameter was extracted from the CME group API
* Address reviews. Adding unit test for failing case
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* fix exchangeinfo downloader
* normalize data, update symbol properties
* min ordersize should be in base currency
* check min order size using quote currency price
* check order size in quote currency
* improve checks
* improve code; more unit tests
* more description
* fix failing test
* Move Aggregate Quote and Trade Bars to utils
* Add decimal and long overload to UnixTimeStampToDateTime
* Add Kraken OrderType check in CanSubmitOrder
* PR !5969 review fixes
* Implement scheduled event sampling solution
* Use UTC time, only update daily portfolio value once a day
* For daily resolutions sample chart always
* Cleanup
* Drop resample daily all together
* Force final sample
* Regression updates
* FIx LiveResultHandler to update portfolio and benchmark values outside of sampling event
* Name the daily sampling event
* Address review pt 1
* Drop force and use reference wrapper
* Adjust tests
* Fix warning for Benchmark Timezone Misalignment and also add test
* Fix for daily resolution orders and test adjustments
* Also warn on universe settings with daily resolution
* Update missed regression
* Fix reference wrapper use
* Update regression after rebase
* Add values back in for Daylight Algo
* Have statistics builder skip day 1 performance
* Regression adjustments
* Test adjustments
* Update regression unit test
* Adjust some regressions starts to show performance values
* Add hourly algorithm for beta comparison
* Address missing Python regression changes
* Remove null comment
* Bitfinex Fee adjustment
- Fix for bitfinex fee adjustment for cash account type only happen if
the currencies are the expected ones, else log message.
* Adjust error handling to terminate algorithm
* Add bitfinex TESTBTCTESTUSD
* Minor improvements
- Add bitfinex test symbols to the SPDB
- Make SecurityDefinitionSymbolResolver log using trace
- Minor tweak for Bitfinex DQH lock contention
* - Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type
- Change CanSubmitOrder() method implementation in GDAXBrokerageModel.cs to get the MinimumOrderSize from the security SymbolProperties directly
- Add CanSubmitOrder() method in Binance, Bitfinex and Kraken Brokerages models
- Add tests to check if CanSubmitOrder() method is working as expected in Binance, Bitfinex, Kraken and GDAX brokerages models
- Add tests to check if MinimumOrderSize parameter is loading correctly from symbol-properties-database.csv
* Revert "- Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type"
This reverts commit e0fd66a5fa.
* Revert "Revert "- Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type""
This reverts commit b0f37935fc.
* Requested changes in Crypto brokerage models
- Remove unnecessary commas in symbols-properties-database.csv
- Add TestHelpers class to reuse GetSecurity() method
- Add IsValidOrderSize() method in DefaultBrokerageModel class
- Remove unnecessary classes for brokerage models test helpers
* Nit change
- Remove unnecessary attribute from DefaultBrokerageModel.cs
* Nit change
* Nit changes and Regression test
- Add Bitfinex BTCUSD hourly data for 2 days
- Add regression test BuyBTCWithLessThanOneDollarAlgorithm.cs to test that the order size is taking into account when placing or updating an order
- Nit changes
* Change CanUpdateOrder() implementation
- CanUpdateOrder() now checks if the requested quantity value is valid
* Requested Changes
- Change regression algorithm BuyBTCWithLessThanOneDollar.cs name and implementation
* Nit changes
* Add BaseDataCollectionAggregatorReader
- Add enumerable for BaseDataCollection
- Add BaseDataCollectionAggregatorReader
- Clean up CollectionSubscriptionDataSourceReader
* Address reviews
- Add new FileFormat that will be handled by the new Reader
- Adding unit tests for the new BaseDataCollectionAggregatorReader
- Some DataSourceReader duplication cleaning up
* Minor adjustment after some live trading tests
Adds Warnings to MOO and MOO using Tick Data
These warnings are meant to let the user know that it was not possible to find ticks marked with the `OfficialOpen`, `OfficialClose`, `OpeningPrints` or `ClosingPrints` flags.
* Fixes Market On Close Fill of Equity Fill Model
Only use trade data (`Tick` with `TickTrade` type or `TradeBar`) to get the closing price, since MOC is filled with the closing price.
- Fix unit tests to show that the new implementation only fills with trade data from the current open market.
- Change regression tests to reflect the bug fix. All other changes are the consequence of using `TradeBar` instead of `QuoteBar`.
* Address Peer-Review
Fixes the logic with tick data: looking for the OfficalClose or ClosingPrints only apply if data from the extended market hours is included.
Adds additional logic to handle missing trade data.
* Refactors Timeout Logic and Adds It to MarketOnOpenFIll
-Refactors timeout logic to address peer-review and adds it to MOO fill too.
- Adds unit tests
* Fix IB futures prices where PriceMagnifier != 1
* Fix deadlock in GetOpenOrdersInternal by moving ConvertOrder call out of event handler;
Minor refactor to combine NormalizePrice and ConvertPrice methods
* IB future and future options price magnifier
* Address review. IB API clean up
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Lean exchanges improvements
- Adding new Exchange class to avoid exchange code clash.
Adding and updating unit test
* Add Market for MapFile API
* Self review
* Rename Renko to ClassicRenko
* Rename WickedRenko to Renko
* Adjust tests and usages
* Backwards support WickedRenkoConsolidator
* Renames for consistency
* Refactor for two types in composite indicator
* Fixes for unit tests
* Non generic indicator base
* Further seperate non-generic indicator base and generic input base
* Adjustments
* Remove generic CompositeIndicator
* Add test suite
* Make each method an individual case
* Address review
* Testing cleanup
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Adds support for getting a Lean symbol based on FIGI, CUSIP, ISIN, SEDOL
* Address review: bug fixes for mapping, improve docs, cleanup & refactor
* Make LocalZipMapFileProvider and LocalZipFactorFileProvider only
initialize if they haven't been initialized yet.
* Address review: makes ISIN, SEDOL, CUSIP case-insensitive
* Cleans up map file provider in SecurityDefinitionSymbolResolver
* Modifies some test cases to test for case-insensitivity
* Add null check for SecurityDefinitions
* Fix unit tests
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add ValueAreaVolumePercentage Parameter
Remove hard coding of parameter and set it to optional parameter for the user.
* Update TimeProfile.cs to include valueAreaVolumePercentage parameter
* Update TimeProfile definition in QCAlgorithm.Indicators.cs
* Create Market Profile indicator.
* - Requested changes in MarketProfile indicator made it.
- TimeProfile and VolumeProfile created and tested.
- tp_datatest.csv and vp_datatest.csv extracted from https://github.com/bfolkens/py-market-profile
* Final potential Market Profile Indicator.
All unit test made it and the indicator is passing all of them
Test cases made it with python library from GH issue
* Code styling request changes
* Minor suggestions
* Minor renaming tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Few improvements
- Alpha Streams algorithm will add target securities right away
- If algorithm is warming up PortfolioTargetCollection will emit not
insight
- Fix local disk factor file provider to use local cache instead of data
folder
- EWAS PCM will not emit targets for securities which have not been
added by the algorithm yet, to avoid runtime exception
* Fix GetLastKnownPrice default order adding more unit tests
As most brokers use separate exchange codes for CBOE and CFE, we added the new Market.CFE (previously Market.CBOE was used for both exchanges). This change also avoids hacky code in symbol mappers.
* Fix zerodha async order event
consider fillQuantity as total filled quantity
cover edge case of fully filled
fix async events received from zerodha
indentation fix
* Fix coding style
* keep order in only CachedOrderIDs
* Alpha Streams improvements
- Warmup added securities so we can trigger an order right away
- Fix bug where AddSecurity returned a security which was not the one
being used
- Fix bug where EWAS PCM would remove from a dictionary while iterating
over it
* Fix unit tests
* Add QCAlgorithm.GetLastKnownPrices API
- Add new GetLastKnownPrices that will return the last known data point
for all subscribed data types.
- Fix for MHDB GetDataTimeZone which was using an invalid entry key
format for custom data.
- Fix for using GetLastKnownPrice/s to seed a security during creation,
when it's not added in the Algorithm.Securities collection
* Adding more unit tests for GetLastKnownPrice
* Address reviews
- Refactor GetLastKnownPrice/s to perform a single history request for
all data types
- After https://github.com/QuantConnect/Lean/pull/5872 trading API
changes numpy float64 was not converted correctly by pythonNet and
used an int. Reverting API changes and adding regression test. This
should be fixed at pythonNet layer
* Adds ETF(...) to UniverseDefinitions
* Adds ETF constituents universe framework regression algorithm
for C#/Python
* Address review: adds test cases for ticker/Symbol ETF universe additions
* Fixes bug where null Market would result in null dereference exception
* Address review: add missing Index tests
* Address review: don't hardcode market when creating constituent universe
* Uses Brokerage Model's default markets collection to determine
the market for the given security type
* Address review: restore QC500 and DollarVolume.Top(...)
* Restores algorithms related to both helper universe
definition methods
* Address review: remove copy to output directory for python algos
* Add example algorithms for ETF constituent universes using custom RSI alpha model
* Address review: adjust algorithm to use cache + algo RSI & clean up code
* Address review: make ETF Constituent RSI Alpha Model algo a regression test
* Address review: increase trade count and remove single trade logic
* Order handling improvements
- Execution model will only trigger market order if they are above the
minimum order margin portfolio percetage value
- SecurityCache.Reset is complete
* Python Import fixes
- Add regression test for ImmediateExecutionModel minimum order margin
check
* Add basic template Atreyu algorithm
- Add C# and Python basic template atreyu algorithm. Show casing how to
specify exchange to execute in different ways.
- Adjust trading API to allow specifying order properties to use
* Lean Exchange improvements
- Rename PrimaryExchange to Exchange
- OrderPropeties will use Exchange enum instead of string
- Adding BSE exchange value
* Regression tests fixes
* To fix the referenced issue
* Fixing up TrailingStopRiskManagementModel
* Fix-ups to address review
* Impl. tests + relevant improvements
* Minor logic improvement at first dictionary update
* Regression test small fix
* Adds explicit 'D' suffix for numbers in test case double arr
* Use integer values in test cases
* Removes failing testcases in MaximumDrawdownPercentPerSecurityTests (!) & renaming
* Revert "Removes failing testcases in MaximumDrawdownPercentPerSecurityTests (!) & renaming"
This reverts commit f9cd279f8c.
* Fix up for failing test cases
* Alpha holdings state
- Alpha result packet will optionally provide the algorithms portfolio
state
* Rename
* Convert AlphaStreamsPortfolio to data source
* Improvements on AlphaStreams algorithm
* Fix regression tests
* Add unit tests for EW AS PCM and fixing bugs
* Protobuf AlphaStreamsPortfoliot staState
- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM
* Add alpha license to Organization response
* Improvements EW AS PCM respects free portfolio value
* Fixes
- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType
* Fixed and adding more regression tests
- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
types
- Add support and regression test for algorithm alpha consumer with existing holdings
* Add AlphaStreamsOrderEvent data type
* More logging for BinanceRestApiClient
* Changing LogLevel to Debug for history providers : Binance, Polygon
* Minor fixes
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Adds support for ETF constituent universes
* Adds filtering for universe data if it doesn't match the
universe subscription type
* Update mapping for ALL underlying Symbols if
`Symbol.UpdateMappedSymbol(...)` is called. Required to support
constituent ETF universes that might have mapping events
* Delistings of composite constituent universe Symbol will result in
removal of universe securities.
* Added regression algorithms for ETF constituent mappings (C#/Python),
along with data required to run locally
* Refactor universe delistings in SubscriptionSynchronizer -
big thank you to @Martin-Molinero :)
* Address review: update regression algorithms and add explanatory comments
* Address review: add additional checks to delisting regression algorithms
* Adds new regression algorithm testing the addition of a universe
without calling AddEquity() and asserts same behavior
* Fix SubscriptionDataReader auxiliary data out of order
- Fix for SubscriptionDataReader emitting auxiliary data out of order
due to sparse that and enumerator refresh logic. Adding regression
algorithm
- Minor tweaks for unit tests failing on and off
* Fix for SDR internal subscriptions
- Fix for SubscriptionDataReader enumerator refresh for internal
subscriptions which were ignored. Adding regression test
- Updating custom data regression algorithms affected by the issue
- DropboxBaseDataUniverse was emitting a custom data point being end time
- UnlinkedTraderBarIconicType was emitting a single data point
of the underlying SPY minute data when if should of emitted
all data points
* websocket-refactor for websocket message as text and binary
Initial commit
websocket support for text and binary messageData
code style fixes
test fix
* delete un-used file
* zerodha websocket textMessage parse
* messageData class abscration
* Empty
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix IB history requests
- Fix time zone bug causing incomplete results
- Use RegularTradingHours flag only for equities
- Second resolution data is now requested in 30 minute chunks (instead of 1 minute):
https://interactivebrokers.github.io/tws-api/historical_limitations.html
* Filter history result respecting market hours and IncludeExtendedMarketHours flag
* Add history unit tests
* Improve python exception parsing
- Improve python exception parsing adding support for line shift. Adding
unit tests
* PythonException revert change
* Centralized and normalize algorithm runtime handling
* Adding support for C# line and file exception report
* Fixes Market On Open Fill of Equity Fill Model
Only use trade data (Tick with Trade type or TradeBar) to get the open price, since MOO is filled with the opening action price. Ensure that this method doesn't use trade data from before the market opens for high-resolution data case.
Fix unit tests to show that the new implementation only fills with trade data from the current open market.
Change regression tests to reflect the bug fix.
In the `ExtendedMarketHoursHistoryRegressionAlgorithm`, MOO was filled with extended market hours.
* Fix Bug for Tick Resolution Case
For tick susbcription, the tick with the open price information is the the first valid (non-zero) tick of trade type from an open market.
Addresses peer-review by moving the if-condition for data belonging to the open market where the subscriscribed types are checked.
* Fix Bug for Low Resolution Edge Case
For the edge case where the order is placed after the trade bar is open, for example, order places at 1 pm with daily-resolution data. The fill model will not use the open of the bar that will close at midnight, since this value is prior to the order.
Adds unit test.
Change regression tests to reflect the bug fix.
In the `RegressionAlgorithm`, MOO was filled with open prior to the order. The algorithm now has one order less, since the last MOO would need to wait another day to be filled.
* Implements SaleCondition and Exchange Check For Tick
- Adds additional unit tests for MOO
* Fixes Regression Test in DataConsolidatorPythonWrapperTests
* Addresses Peer-Review
- Adds new unit test cases.
* Update Lean Foundation docker files to IBGateway v985
* Add apt-get update for ARM foundation
* Update default value of IB gateway
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* WIP removal of custom data references & tests updates
* Regression algos updated and python algorithms moved to DataSource repos
* Fixes failing unit tests
* Add "LiveDataTypes" field to LiveNodePacket
* Adds Initialize() to IDataChannelProvider
* Adds new extension method to convert
HistoryRequest -> SubscriptionDataConfig
* Address review: Add protobuf definitions for Iconic data types
* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo
* Small adjustments to variable naming and documentation
* Move test files to respective DataSource repos
* Add support for IBGateway v985
- Effective in TWS version 985 and later, for US stocks the bid, ask, and last size quotes are shown in shares (not in lots).
* Add missing error log in IB symbol mapper
* Trigger build
- Create Lean common engine initialization class, which will be called
by research and Lean launcher
- Fix backtest deserialization bug due to failing to handle decimal
value
* Refactor pandas mapper to support newer versions
* Clean up remapper
* Finalize deprecated tests and adjust those that should apply to new pandas
* Move mapper to its own file
* Add supporting Py tests and setup instructions
* Add license
- Calling RemoveSecurity() will remove the security from all the
universes holdings it, there was a race condition here where it would
just take the first universe and remove it from it.
Adding regression test adding and removing an option contract and it's underlying
- Take lock to refresh client to avoid null reference
exception
- Reduce gate rate
- Use extension methods to await tasks
- Avoid copying the data multiple times
* Update BinanceBrokerage to handle more than 512 symbols
* Address review
- fetch symbol weights only if required
- remove code duplication
* Add rate limiting for new connections
* Update WebSocketMessage to include the websocket instance
* Handle resubscriptions on reconnect
* Address review
* Address review
* Remove unnecessary locking
* WebSocketClientWrapper updates
- remove allocation of receive buffer on each message
- add missing lock in Close method
- log message data when message type is Close
- fix race condition after unexpected websocket close
* Set WebSocketClientWrapper task to LongRunning
* Add missing check in GetHistory
* Fix exceptions with Binance downloader
- closes#5794
* Update BinanceBrokerage to handle more than 512 symbols
* Address review
- fetch symbol weights only if required
- remove code duplication
* Add rate limiting for new connections
* Update WebSocketMessage to include the websocket instance
* Handle resubscriptions on reconnect
* Address review
* Address review
* Remove unnecessary locking
* WebSocketClientWrapper updates
- remove allocation of receive buffer on each message
- add missing lock in Close method
- log message data when message type is Close
- fix race condition after unexpected websocket close
* Set WebSocketClientWrapper task to LongRunning
* Add missing check in GetHistory
* Fix exceptions with Binance downloader
- closes#5794
* Update Bitfinex symbols in symbol properties database
* Update BitfinexBrokerage to use BrokerageMultiWebSocketSubscriptionManager
* Address review
* Remove unnecessary locking
* Remove old channels on resubscription
* logging and kite connection retry
(cherry picked from commit 9845a2d79e86c338093cadcae7f41b29d897c78c)
* changed thread.sleep time for kite connection retries
* websresponse retry on error - coding style change
* datetome parse to convert time in utc
* indentation code-style fix
* zerodhaDownloader support for dataTimeZone
* removed code redundancy for single conversion without hardcoded TZ
* removed hardcode dataTimeZone and code refactor
* returns exchange TZ to lean and data TZ to local disk from downloader
* Kite historical candle timestamp TZ conversion
* added unit test for kite historical candle
* access modifier correection
* Filter out small orders based on Setting
- BuyingPowerModel will filter out small orders based on algorithm
setting, a % of PTV, instead of hard coded 1 share value. Addin unit
and regression tests
- Updating regression algorithms to use new setting, reduce order trades
* Update regression algorithms
* zerodhaDownloader support for India Market
(cherry picked from commit c36272ef9c4c7eddfff77413d900faab3a2a8654)
* indenation styling fix
* zerodha donwloader
* Market.India support from zerodhaSymbolMapper
* code styling changes
* re adding KiteProductType
* check kiteProductType per order and get all Product Type Account Holdings
* use WithLockedStream and remove lock unlock stream
* filter holding if config productType set otherwise fetch all
* improvements in styling and properties
* unionbank has less margin requirements, second always works
- TransactionHandler will see invalid OrderEvent for an order which
tries to short an non shortable asset. Adding unit test, updating
regression algorithm.
- Fix Api data price path normalization, found while testing with
shortable provider. Adding unit tests
- Improve readme file, add nuget link, update Lean home page
- Increase 10min look back to 60 for setting currency conversion
- Replace error message for trace when minute resolution history request
is empty
* Added Augen Price Spike Files
* AugenPriceSpike Update
Adjusted indicator calculations and added test data from Trading View
* Requested Changes
* Minor changes
* Add reproducing unit test
* Apply fix to force adjustment if needed
* Adjust fee to correctly replicate original issue
* Adjust solution
* Update unit test to reflect the way this function is used
* nit - rename test
* Cleanup error message to improve reproducibility
* nit - Drop unneeded var
* Add in adjustment test cases as well, fixed
* Fix minor race condition in brokerage message handler
- Fix race condition where a message could be enqueued and
wait for a new call, not being processed ASAP
* Reduce BrokerageTransactionHandler logs
* Fix coarse universe generator on single processor
* Make sure MaxDegreeOfParallelism value is not less than 1
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add BrokerageConcurrentMessageHandler
- Add Brokerage helper `BrokerageConcurrentMessageHandler` class to lock message stream
while executing an action, for example placing an order, adding unit tests.
- Binance will use new message handler
* Fix unit test
* Implement solution with lightest changes possible
* Update regressions v1
* Adjust AlgorithmTradingTests
* Adjust PatternDayTradingMarginBuyingPowerModel tests
* Drop need to loop twice
* Adjust last unit test, with calculations included
* nit - comment fix
* Break out adjustment calculation to static function; add unit test
* Update Py regression
* Upgrade adjustment calculation to be smart enough to get us to target always
* nit - cleanup GetAmountToOrder
* Add license to test
* nit - comment fix
* cleanup GetAmountToOrder further
* Add additional test cases
* added symbolData class to store exchange and token values of tickers
* support for India Market
(cherry picked from commit 5f629dcb9e72426dc7b560e782cb1fae72861574)
* removed local vscode workspace file
* support for india market instead of nse and other exchanges
* changed hardcoded india values to Market.India
* changed zerodha getcash fromm available to net
* addded edge case handoing with expections
* support for market info (nse, bse) for india market
* exchange info for EmitQuoteTick
* basic testing with c#
* c# algo
* c# testing
* live orders during market open success
* support orderproperties for exchange value for order placement
* added exceptional handling cases
* fix xml descriptions for the files
* fix zerodha-product-type in DefaultOrderProperties
* add basic template for india market
* fix orderProperites for default values
* fix for local data, python implementation not available
* fix access modifiers of SymbolData
* fix variable name case
* fix single check for exchange, call to base
* fix styling, redundent code and access modifers
* fix non static not callable from static
* fix static
* Adjust AlgoSeek Futures price multipliers to quote prices in USD
* Revert changes to AUP scaling
* After a second review, seems that this contract isn't scaled.
The contract multiplier differs from the NYMEX Aluminum Index's
contract multiplier, and so does the minimum price variation.
The minimum price fluctuation seems consistent with CME's
quoted prices, which also claims the contract is quoted in USD.
* Fix futures contract multipliers for some scaled contracts
- Add new CompositeDataProvider that will wrap multiple instance. Allows
using for example the ApiDataProvider and a DownloaderDataProvider.
- Adjusting ProductJsonConverter after recent Modules addition
* Add new DownloaderDataProvider
- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
IBrokerage implementation as data source
* Add support for downloading symbol chains data
The Kraken allow queries from pairs of format XBTUSD and XXBTZUSD but it will only return data with the XXBTZUSD causing errors when XBTUSD ticker is specified.
This commit adds a more meaningful error when KDL is called with XBTUSD format.
* Update IBAutomater to v2.0.39
* Downgrade IB logging level from Detail (5) to Information (4)
- The online documentation warns of performance overhead when setting logLevel=5
* Apply fix for both start and end month cases
* Add unit tests for both cases
* Fix weekstart/weekend for same issue; add unit tests
* nit - rename tests
* Adjust logic
* Update charts that are not empty and not default values
* Testing
* Filter final results
* Explicitly filter alpha charts if not needed
* Use filtered charts for Live results handler
* Refactor solution to manage alpha charts internally by creator
* nit comment fixes
* Implement drawdown as default chart
* Implement Capacity Estimate for backtest default chart
* Add default values to QC.Plotting
* Adjust drawdown calculation
* Only sample "Capacity" once a day
* Implement Volume chart
* Nit fixes
* Volume Chart adjustments and fixes
* Rename to "Assets Sales Volume"
* Implement Exposure Sampling
* Store algorithm currency symbol, round capacity to nearest 1k
* Add new plots to default charts
* nit change name
* Address reviews
* Reduce duplication and clean up exposure sampling
* Address reviews
* Improve sample exposure
* Address review
* Only enumerate holdings once
* nit - comments
* Post rebase fix
* Don't need to round anymore
* SymbolJsonConverter improvements
- SymbolJsonConverter will be able to deserialize symbols expressed in
user friendly format. Adding unit test
- Adding a couple of helper methods, removing duplicated code
* Fix FOPs expiration year parsing
* Add IResultHandler.BrokerageMessage
- Result handlers can receive notifications for brokerage message events
* Make LiveTradingResultHandler.RuntimeError virtual
- Reduce thread contention in coinApi DQH by taking a lock per symbol
versus a global lock.
- Improve tick creating by reducing object being wasted on empty
constructor
* Updates Equity Market Data
* Updates Unit Tests
* Updates Regression Tests
In this commit we include regression tests with small changes (slightly different CAGR, Alpha, etc, but same number of trades) due to the data update.
* Updates Regression Tests 2
The following regression tests were adapt because of verification of hard-coded market data values:
- `AdjustedVolumeRegressionAlgorithm`
- `HistoryWithSymbolChangesRegressionAlgorithm`
- `OptionRenameRegressionAlgorithm`
- `RawDataRegressionAlgorithm`
- `SwitchDataModeRegressionAlgorithm`
The following regression tests have more trades since adjusted prices allowed more 1-2 shares trades that were rounded down to zero before:
- `AddUniverseSelectionModelCoarseAlgorithm` 23 -> 35
- `MeanVarianceOptimizationFrameworkAlgorithm` 12 -> 14
- `PortfolioRebalanceOnDateRulesRegressionAlgorithm` 298 -> 324
- `PortfolioRebalanceOnInsightChangesRegressionAlgorithm` 83 -> 86
- `ScheduledUniverseSelectionModelRegressionAlgorithm` 86 -> 90
- `SectorExposureRiskFrameworkAlgorithm` 17 -> 22
- `SetHoldingsMultipleTargetsRegressionAlgorithm` 8 -> 9
- `StandardDeviationExecutionModelRegressionAlgorithm` 196 -> 199
- `UniverseUnchangedRegressionAlgorithm` 11 -> 17
- `VolumeWeightedAveragePriceExecutionModelRegressionAlgorithm` 237 -> 238
Especial cases:
- `BlackLittermanPortfolioOptimizationFrameworkAlgorithm` 18 -> 17
- BLM model sensibility
- `OptionChainedAndUniverseSelectionRegressionAlgorithm`
The following regression tests have different Capacity because of different volume from lowest capacity asset, except:
- `OptionEquityCoveredCallRegressionAlgorithm` New lowest capacity asset is underlying
- `OptionEquityCoveredPutRegressionAlgorithm` New lowest capacity asset is underlying
* Revert File Update for SPWR and SPWRA
* Fix Regression Tests
Temporarily removes python regression test for `MeanVarianceOptimizationFrameworkAlgorithm` as the `MeanVarianceOptimizationPortfolioConstructionModel` for each version are yeilding different results. If we use C# version in `MeanVarianceOptimizationPortfolioConstructionModel.py`, the results match.
* Changes Optimization Method in MinimumVariancePortfolioOptimizer [Py]
Uses `trust-constr` method.
See https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html
* Use Python algorithms directly in their location; also refactor pythonpath additions to one method
* Update VSCode Launch options
* Drop issue these changes fixes
* Use AddRange
* Python research import improvements
- Improve start.py for research env
- Remove unrequired imports
* Centralize algorithm imports
* Add regression test GH action
* Unit test python import clean up
* Join research and main imports
* More python import clean up
* Fix failing skipped regression algorithm
* Stop update when properties will fail to update
* Only enforce our case after the BB is ready
* Nit comment fixes
* Add unit test
* Use faster DateTime.UtcNow
* Updates to default notebooks
* Update readme
* Notebook adjustments
* Drop launch script support
* Use our cwd if no QC dlls are in AppDomain directory
* Modify Intialize.csx to use parent directory only if QC files are detected there
* Clean up some details after testing
* Adjustments
* nit, NullOrWhiteSpace
* Enable loading live virtual position tracking
- Enable GDAX paper trading configurations
- GDAX will optionally load existing holdings from live job package
- Bitfinex will try loading existing holdings from live job package for
cash account type
* Binance live holdings loading
* Adds VIXCentral Contango data
* Refactor cached alternative data in live mode to
use web endpoint if the data is not available locally
* Remove fallback to cache bucket for CBOE/FRED/EIA/VIXCentralContango
* Address review: force VIX as ticker for all VIXCentralContango data
* Normalize live holdings initialization
* PaperBrokerage config based live state recovery
- Increase algorithm status storing sampling period
- Fix bug where duplicate cash entries could happen
* Applies a fix
* Fixing the tests to cover downloader func. Fixing the queue handler
* this has to be removed
* -- Fixes -- [WIP]
* Implements the way to upload historical bars in async fashion. [ WIP ]
* Finishes GetEquityTradeTicks() impl.
* Do not break iteration if result is null
* Uses DisposeSafely and fixes a typo
* To use IEnumerable for return type in GetHistoryAsync
* Fixes GetEquityQuoteTicks()
* Removes GetHistoryAsync() and reverts the associated with it changes
* SubscriptionManager -> revert the changes
* PolygonDataQueueHandler -> use SIP timestamp as it keeps the order
* PolygonDataQueueHandler -> use Log.Trace for the no data {date} message
* Fixes GetForexQuoteTicks()
* Introduce ResponseSizeLimitAggregateData
* Fixes up GetCryptoTradeTicks()
* Small fixes for tick processing methods
* Removes launchSettings.json
* Refactor download data logic
- Replace WebClient for HttpClient
- Remove duplicated download logic and parsing for a single static
method
* Update PolygonDataQueueHandler.cs
Remove commented code
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Use DataProvider for fetching Shortable and FutureMargins files
* Use data provider for AllShortableSymbols
* Refactor of NeedToDownload logic
* Adjustments
* nit - cleanup
* Expand test cases
* Create the directory if it doesn't exist
* Add immediate logging for IB brokerage unit tests
* Filter brokerage message event for IB warning
- 10197 - No market data during competing live session.
* Update IBAutomater to v2.0.32
* Update IBAutomater to v2.0.33
* Use determined quantity for exercise order
* Add regression algorithm for issue
* Minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Ensure data is downloaded only once with the ApiDataProvider. Adding
unit test
- Log invalid securityType ApiDataProvider requests once
- Assert Factor file and map file providers are compatible with the
ApiDataProvider
* Use lean data key as param for request
* key -> filePath rename and some cleanup
* Refactor
* Add Organizations Endpoints
* Add some organization api wrapper objects
* Address namespace issue
* Reorganize Api Test into seperate files using one ApiTestBase
* Add Organization tests
* Use capitalized "API" test namespace to reduce amount of file changes
* Add License to test base
* Update /data endpoint functions and response objects
* Update ApiDataProvider Logic
* Handle deserialization of organization products
* Simplify converter
* Only throw for equity requests when not subscribed to map/factor files
* Add missing header
* Make arguement exception
* Api adjustments
* Add Zip factor and map file providers
- Common project will now reference Compression project and not the other way
round.
- Adding Zip FactorFile and MapFile providers
* Refactor FactorFileProvider to use DataProvider to fetch files
* Use resulting MinimumDate in construction of FactorFile
* Nit FactorFile comments and arrangement
* Refactor MapFileProviders to use DataProvider for fetching files
* Refactor ZipFileProvider
* Clean up
* Refactor Backtesting Future/Option chain providers to use dataprovider
* Fixes for data/ endpoints and test adjustments
* Response objects adjustments/cleanups
* ApiDateProvider fixes and testing
* Add LocalZipFactorFileTests
* Update ApiDataProvider download test to verify stream is not null
* Implement posting of agreement summary and signed time
* Mark all Api related tests as explicit and document details on running
* Clarify default token on ApiTestBase
* Adjust summary
* Update Api responses for QCC, except org products which are sold in USD
* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests
* Fix multiple markets for ZipFactorFile provider
* Use Symbol as cache key
* Api.cs review
* Dispose of factorFileStream after reading
* Use zip.EntryFileNames
* Address a few reviews
* Few more fixes
* Address Api Review
* Add Job Org id to config
* Minor tweaks
* Compare with invariant culture
* Fixes Option Universe selection
* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider
* Fix research
* Fix null reference exception
* Make duplicate log debug
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fixes bug where raw data was incorrectly be filtered when ID was duped
* Filter CoinAPI raw data by distinct Symbol created from the SymbolMapper
* Fixes bug where only quotes or trades would be processed, not both
* Trigger rebuild
* Attempt to remove SecurityInitializer from UniverseSelection stack entirely
* Fix broken Python Universe Selection model
* Remove mistake change (meant for another branch)
* FOPs margin as a curve
- Adjust FOPs margin requirement so that it behaves like an 'S' curve
for the contract strike, around the underlying price and it's margin requirement.
Based on IB behavior. Adding unit tests for different future contracts
- Improve a couple of unit test cleanup
* Add ES Short Call unit test
* Adds filtering of negative ticks in CoinAPI converter
* Tick data will remain the same, only aggregated data
will have changes reflected in its output as a result
* Adds suspicious tick filtering to ticks
* Address review: Add Crypto Trade Tick suspicious test case
* Implements PivotPointsHighLow indicator + tests
# Conflicts:
# Tests/QuantConnect.Tests.csproj
* Some fixes
* Refactoring + Adds NewPivotPointFormed event
* Fixing tests
* Some fixes for the Get methods to not throw when points array is empty
* To address review + more fixes
* Fixing xml comments typos
* Implements separate rolling windows to calculate highs and low
* Makes the number of last stored indicator values as an input parameter
* Overrides Reset()
* Adds a helper method
* Change numerical return to Enum types
* Changes IsReady condition :
the indicator is ready and starts calculating the pivot point when any of the rollings is ready
* Address reviews
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Update launch options and tasks
* Update readme
* Mention building manually is optional
* Add portable pdbs to use C# coreclr debugging
* Update VS and VSCode docs
* Add reference to Lean CLI in main readme
* remove run_docker scripts
* Remove PyCharm files
* Cleanup
* Remove run_docker tasks from VS Code
* Address typos
* Implemented AlphaVantageDataDownloader
* Style and documentation changes requested
* Throw exception for unexpected response content
* Added rate limiting to AlphaVantageDataDownloader
* Minor tweaks and rebase
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Apply changes from #2146 on current master
Credit for most of these changes goes to https://github.com/viliwonka.
Rebasing three years of changes didn't go so well, so I manually
applied the changes in #2146 on the current master branch.
This commit attempts to represent the changes in #2146 as closely as
possible, no cleanup has been performed yet. Despite this, some changes
were inevitable because part of #2146 has been implemented another way
in the past three years.
* Clean up
* Process review comments
* Add tests
* symbol -> potentialConversionRateSymbol
* Ensure regression tests run/get data
* Fix broken test
* Process review comments
* Process review comments
* Simplify currency conversion logic
* Performance improvement, doc updates and test rename
* Rebase fixes
* Remove obsolete properties
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Minor tweaks
- Add AccountCurrency and symbol for live result packet
- Improve performance of Tick clone and reading
* Reduce unit test from 9s to 500ms
* Do not restrict unrequested securities by 1.0m leverage, let security initializer do its job
* Use UniverseSettings for leverage, fillforward, and extendedmarkethours
* Update unrequested security message
* Use UniverseSettings ExtendedMarketHours for underlying on OptionContracts
* Add unit test to prove setting persistence
* Cleanup unit test
* Adjust crypto unit test assert
Co-authored-by: Colton Sellers <Colton.R.Sellers@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix FutureOptionMarginModel Margin Requirements
- Fixes FuturesOptionsMarginModel margin requirements being zero due to
underlying being null on model creation. Adding unit test
* Address reviews
- SecurityService will set underlying when creating a new security when
provided
* Fixes SEC rate limiting issue by reducing clustering of requests
* We add a user-agent header to all outgoing requests for SEC
as they require it to identify the client, otherwise we're rate
limited.
* Some adjustments to how we wait for work to be done, and RateGate
rate of requests. We wait for all requests now/don't use tasks
anymore
* Address review - remove class HttpClient in favor of local instance
* Upgrade IBAutomater to v2.0.0
* Upgrade IBAutomater to v2.0.1
* Upgrade IBAutomater to v2.0.6
* Upgrade IBAutomater to v2.0.7
* Upgrade IBAutomater to v2.0.9
* Upgrade IBAutomater to v2.0.10
* Upgrade IBAutomater to v2.0.12
* Upgrade IBAutomater to v2.0.13
* Upgrade IBAutomater to v2.0.14
* Upgrade IBAutomater to v2.0.15
* Reset disconnected flag on IBGateway exit/restart
This bug was causing invalid algorithm termination after reconnection
* Fix unhandled exceptions in exit/restart event handlers
* Upgrade IBAutomater to v2.0.16
* Upgrade IBAutomater to v2.0.17
* Update testing OrderProvider to clone orders as BrokerageTransactionHandler
* Add immediate logging for brokerage unit tests
* Tradier brokerage updates
- Fixed NullReferenceException with zero-crossing orders
- Added a few missing null checks
- Fixed issues in error handling
- Added new unit tests for some common errors
* Adds processed data directory to read price data from
* Make coarse universe generator look at data directory before failing to find daily data
* Set coarse generator output of missing daily file to debug log
* Add CoarseUniverseGenerator logs
* Fixes 100 nanosecond increment lookahead bias when parsing large numbers
* Whenever we parse a number that is has precision greater than
DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
between [0, 1000), excluding numbers divisible by 100,
we will have leftover nanoseconds between [0, 100) nanoseconds, but
they won't be factored in to the DateTime calculation, since casting
to `long` only takes the integer component of the number, so we lose
the extra nanoseconds that came with the decimal, and time is set to
the "floored" value without those nanoseconds.
Since .NET `DateTime` type has a limitation of only being able
to represent time in increments of 100 nanoseconds, by not
considering the sub-100 nanoseconds, we introduce a look-ahead
bias of at most 100 nanoseconds/1 tick
* Misc adjustment to make method use `decimal` instead of `double`
for increased precision when parsing large numbers
* Changes CoinAPI data converter to support processing raw files in original directory structure and file name
* Removes Market requirement from CoinAPI data converter
* Remove timeout on decompression of raw AlgoSeek futures data
* Updates SEC downloader to use HttpClient where requests were failing
* For some unknown reason, valid requests to a valid URL were
failing when using WebClient. Changing our requester to
HttpClient fixes the issue, and enables us to leverage
async capabilities where applicable.
* Added fault tolerance to index file downloads, including a
rate limit in case we've been rate limited
* Further refactoring; catches 429 errors, adds missing rategate calls
* Replace all usage of WebClient, force retry for all failures
* Adds optional config value for Benzinga News API key in downloader
* Modifies Estimize Downloader api config name and fixes directory not found bug
* Refactor Estimize to speed up processing time
* Adds ticker limits if desired
* Misc. bug fixes, performance improvements, code cleanup
* Remove debug log statements leftover from previous commit
* Add support for non-tick Index resolutions in LeanDataWriter
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Lower requests/second for SEC downloader, add missing rategate call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Fix FOPs maintenance margin requirements since they were relaying on
the underlyings holdings cost to return 0 in some cases. Updating unit
tests
- Fix Option exercise order being rejected because of margin not
available, will now take into account the margin used by the option
itself. Adding unit test
* Add property for capacity. Remove unused variable
* Move SymbolCapacity and CapacityEstimate to common, passed through Symbol to runtime statistics
* Add null checks
* Remove uninvested and untradable assets from capacty calculations
* Add SymbolCapacity influential period
* Updates Regression Tests
- DelistingEventsAlgorithm
- Allows additional contributions from delisted AAA.1
- DelistingFutureOptionRegressionAlgorithm
- Removes DC01H12 contributions one month later
- FutureOptionBuySellCallIntradayRegressionAlgorithm
- Allows additional contributions from future after expiry replacing the contribution of the next contract option
- DelistedFutureLiquidateRegressionAlgorithm
- FutureOptionCallITMExpiryRegressionAlgorithm
- FutureOptionCallITMGreeksExpiryRegressionAlgorithm
- FutureOptionPutITMExpiryRegressionAlgorithm
- FutureOptionShortCallITMExpiryRegressionAlgorithm
- FutureOptionShortPutITMExpiryRegressionAlgorithm
- FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm
- Allows additional contributions from future after expiry
- FutureOptionCallOTMExpiryRegressionAlgorithm
- FutureOptionPutOTMExpiryRegressionAlgorithm
- FutureOptionShortPutOTMExpiryRegressionAlgorithm
- IndexOptionCallITMGreeksExpiryRegressionAlgorithm
- IndexOptionCallOTMExpiryRegressionAlgorithm
- IndexOptionShortCallOTMExpiryRegressionAlgorithm
- Allows additional contributions from option after expiry
- MACDTrendAlgorithm
- Removes contribution when SPY is not invested for over one month
- UniverseSelectionRegressionAlgorithm
- Allows additional contributions from delisted GOOAV replacing GOOG (new symbols)
* Adds Lowest Capacity Asset to Regression Tests
* Normalize expected value -0, because -0 is also written to file if updated
* Write Symbol.Value for lowestCapacitySymbol or empty string for empty Symbol
* Update Regressions
* Update 'Lowest Capacity Asset' to Symbol.ID
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Co-authored-by: Colton Sellers <Colton.R.Sellers@gmail.com>
* Updates Fundamental Data
* Updates Regression Algorithms
- `CoarseFundamentalTop3Algorithm`
- New coarse has higher `DollarVolume` for `FB`
- `CoarseNoLookAheadBiasAlgorithm`
- Change from `SPY` update not included in #5493
- `SectorExposureRiskFrameworkAlgorithm`
- `HasFundamentalData` was false for `GOOG` on 20140401 and 20140402.
* Updates Unit Tests
Minor changes in expected values
* Updates Regression Tests
Should have been included in #5493:
- `OptionsExpiredContractRegression`
- `FuturesExpiredContractRegression`
* Adjusted Quantity By Lot Size in OrderTargetsByMarginImpact
`OrderTargetsByMarginImpact` didn't calculate the order value with the quantity adjusted by order size which less to values that did not reflect the actual order value.
It has a particular affect in `SectorExposureRiskFrameworkAlgorithm` where the Python and C# versions have the same orders but placed in a different sequence because of decimal/double precision.
* Reduce dictionary access x3 on OrderTargetsByMarginImpact
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Redefine IsReady and WarmUpPeriod
* Update warm up periods of other indicators to account for ROC warmup change
* Update regressions affected by this change (because of HistoricalReturnsAlphaModel using ROC)
* Add simple unit test for issue #5491
* Cleanup warmup period math and add comments
Co-authored-by: Colton Sellers <Colton.R.Sellers@gmail.com>
* Reconcile and consolidate SetupHandlers
* Centralize GetConfiguredDataFeeds()
* nit - extra space
* Check for null, also allow null to be passed back if no config value
* Fix breaking test
* Cleanup
* fix return var
* remove unneeded if/else
* Minor changes
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Use UniverseSettings.DataNormalizationMode for securities added in Algorithm
* Stop SubscriptionUtils from forcing Adjusted mode
* Return behavior to original and add comments
* nit typo
* Add regression
* Add unit test that verifies DataNormalizationMode can be altered manually by security
* Cleanup and add Py version of regression
* Update projects to use .NET 5.0, the successor to .NET Core
* Fix ambiguous errors. Add IBAutomator net5
* Remove FXCM
* Upgrade IBAutomater to v1.0.51
ignored, and an empty message aborts the commit.
* Fix rebase
- Fix ambiguous Index
- Remove StrategyCapacity.cs
- Update System.Threading.Tasks.Extensionsy
* Remove unrequired references
* Fixes
- Travis will use dotnet, not nunit nor mono
- Remove mono from foundation image
- Fix python setup in research
- Fix unit tests
* Don't call ReadKey when input is redirected
* Fix ConsoleLeanOptimizer
* Research fixes
* Update comment
* Add vsdbg to Dockerfile
* Fixes
- Revert dockerfile FROM custom changes
- Adjust and fix regression algorithms
- Option assignment will be deterministic in the order
- 'Rolling Averaged Population' is calculated using doubles, updating
expected values.
- Update readme, removing references to mono
- Add missing Py.Gil lock
* Replace ICSharp with .NET Interactive
* Fixes after rebase
* CSharp research fixes
- Adding new Initialize.csx that pre loads all assemblies
- Adjusting template research file
- Moving steps in dockerfilejupyter
- Fix unit tests and regression tests after rebase
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Jasper van Merle <jaspervmerle@gmail.com>
* Add unit test
* Make test cover issue case
* Filter out times before startTimeUtc
* Refactor solution, fixes missing first date
* Fix case where none is expected
* Cleanup tests, add TriggerTimesNone
* Drop unused imports
* Add missing dipose call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Refactor HasSufficientBuyingPowerForOrder implementations
Adds Sufficient and Insufficient helper methods to HashSufficientbuyingPowerForOrderParameters
enabling syntax like:
return paraeeters.Sufficient()
returnparameters.Insufficient(reason)
The next change will add the initial margin required which will simply require
updating both of these helper methods to accept the value.
* IBuyingPowerModel: Add margin functions Maintenance/Initial/ForOrder
These were originally hidden in an effort to only expose what's necessary
for the engine to perform its work. Additionally, we encapsulated all of
the method arguments into parameters classes to prevent having to break
anyone in the future. Not including these foundational methods turns out to
be an oversight. These methods are not required by the engine, but rather by
other models. Another possible solution here is to add an additional abstraction
and include these methods on this new abstraction. BuyingPowerModel would then
explicitly implement these methods and models that depend on them would require
two code paths, one for when the buying power model implements this interface
and another for when it doesn't.
Tests were additionally updated to remove test model implementations created for
the sole purpose of exposing these private methods.
* Add ConstantBuyingPowerModel
Provides an implementation of IBuyingPowerModel that returns the same
constant value
* Update BuyingPowerModelPythonWrapper to use reflection for method names
Having a bunch of hard-coded strings is a sure fire way for someone to
overlook when changing methods. This change ensures that noone needs to
remember that this code exists :)
Cleans up the syntax around verifying a python object implements a particular
C# interface via the ValidateImplementationOf<T> method by having it return a
value since the only use cases are in constructors when setting the models.
I was initially going to update ALL python wrappers to validate the passed
in models, but such a change could break many things that are 'working' right
now. Such an effort should be saved for its own dedicated PR.
* Add Parameters/Result types for new buying power model methods
* Support computing maintenance margin for arbitrary quantities
The existing GetMaintenanceMargin function assumes that we're only interested
in the maintenance margin for the entirety of the provided security's holdings.
This makes it impossible to perform what-if analysis or to even ask how much
maintenance margin is devoted to a particular subset of the security's holdings.
This change adds the quantity to the MaintenanceMarginParameters class. Futures
and Options models also depend on holdings cost and holdings value, so they have
also been added to the parameters type. Finally, static factory methods were
added to improve discernment of intent: ForCurrentHoldings provides the existing
behavior and then ForQuantityAtCurrentPrice to support what-if scenarios where
we're looking for the change in maintenance margin if we were to execute an order
for the securiy at the current time step. Obviously a constructor is provided to
set all of the values explicitly, using any price metric the caller desires.
* Address review
- Fix BPM xml documentation
- Fix python unit tests and PythonWrapper validate method
* Add SecurityHolding.QuantityChanged event
Adding event handlers will allow us to orchestrate complex
events from distant parts of the codebase through wiring
them up. If we continue down this path, it will move us away
from the current, very 'mechanical' data flows expressed in
LEAN and towards a more modern, event processing based system.
This is but a baby step in that direction and the initial use
case is using this QuantityChanged event to trigger resolution
of the algoritm's positions groups. This is part of an effort
to improve the fidelity of options margin modeling where we'll
model an OptionStrategy as an IPositionGroup. This will allow
us to compute the margin requirements of an OptionStrategy as
a unit instead of computing margin of each security individually
in isolation.
See #4065
* PortfolioManager: Group fields and remove unused field
This codebase generally places fields as the first members, but
this class had some fields at the top, then some properties, and
then some more fields. This change brings all the fields together
at the top of the file and also removes pointless comments placed
directly above some of the fields. Additionally, an unused field
was removed.
* Remove unused _currencyConverter from Security
Looks like at some point the only code using this member variable was removed
and the necessary clean up was overlooked.
* Add Parse.Enum functions
* Support disabling regression algorithms by language via config.json
Adds 'regression-test-languages' to config.json and filters regerssion algorithms to
run based on this value. When cycling on a particular feature, it's nice to be able
to run the entire regression set while ignoring the python algorithms. Once the C#
algorithms are all passing, one can then go back and run C# and Python in a final run,
since 99% of feature work doesn't impact python specifically.
* Implement IComparable in SecurityIdentitfier
This can be used to deterministically sort securities and symbols
* Add .editorconfig to enforce common formatting for json/sh files
* Fix typo in IBuyingPowerModel.GetBuyingPower xml docs
* Add ListEquals/GetListHashCode and OrderDirection.Closes(PositionSide)
ListEquals and GetListHashCode are designed to be used together as they
complement each other according to C#'s requirements for Equals and
GetHashCode functions.
PositionSide.ToOrderDirection() extension simply converts a PositionSide
to its logical equivalent OrderDirection. Long->Buy, Short->Sell, None->Hold
OrderDirection.Closes(PositionSide) determines if a particular OrderDirection
would have the effect of reducing a position's absolute size. This function
greatly improves the readability of buying power functions that must provide
adjustments when an order/contemplated trade reduces/closes an existing position.
OrderDirection.Buy.Closes(PositionSide.Short)
OrderDirection.Sell.Closes(PositionSide.Long)
All other combinations return false
Adds ToArray/ToImmutableArray convenience functions that combine a call
to Select followed by To(Immutable)Array all in one function call.
* Add decimal.DiscretelyRoundBy extension method
Supports rounding a decimal value by an arbitrarily chosen maximum precision,
or 'quanta'
* Update FutureMarginBuyingPowerModelTests to respect the security's lot size
* Add core position group classes and abstractions
* Add initial/maintenance margin support, buying power model consistency tests
* Add SufficientBuyingPower and GetReservedBuyingPower to position group model
Includes update to BrokerageTransactionHandler to use position group BPM for
sufficient buying power checks.
* Resolve position groups on each fill
We need to update the state of our position groups on each fill so that
we can properly handle multiple orders within the same time step. We
also limit the number of positions sent into the resolver by removing
securities without any holdings.
* fixup! Add SufficientBuyingPower and GetReservedBuyingPower to position group model
* Add GetMaximumLotsFor{Target|Delta}BuyingPower
Instead of computing order quantity, these functions compute the
maximum number of position group lots, which is the position group
quantity, and is guaranteed to be a whole number, for the provided
target/delta buying power parameters.
The SecurityPositionGroupBuyingPowerModel delegates to the security's
IBuyingPowerModel by applying a scaling factor equal to the security's
lot size.
This change also updates references to IBuyingPowerModel.GetMaximum...
to use the new position group model methods.
* Convert remaining IBuyingPowerModel call sites to position groups
* Rename PositionManasger.CreateDefaultGroup -> GetOrCreateDefaultGroup
Better describes its behavior
* Add Position Groups readme.md
* Add Option Strategy BuyingPowerModel
- Adding CompositePrositionGroupResolver and
OptionStrategyPositionGroupResolver
- Adding OptionStrategyPositionGroupBuyingPowerModel handling option
strategies based on IBs margin table. Adding regression algorithms
- Few changes so that option strategies executed by multiple orders are
detected
- Adjust OptionStrategyDefinitionMatch to include equity legs in the
matching result
- Minor tweaks fixing previous rebase
- Minor fixes for existing option strategies definitions, adding new
missing strategies.
- Fixing minor bugs in option strategy matcher. Adding more unit tests
* Address self reviews
- Fixing bug in 'PositionGroupCollection'
- Few minor simplificaitons
- Adding BasicTemplateOptionEquityStrategyAlgorithm
* Address reviews
- Improve regression algorithms margin remaining and used assert logic to be exact. Taking into account spread and fees
Co-authored-by: Michael Handschuh <mhandschuh@gmail.com>
* Refactor to queue up start date changes to log on dispose
* Fix typo in warning collection used
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Collect and log symbols that had start date adjusted because of factor files
* Only post the message if the set has values
* Modify message to be [Symbol, Date] combo for more information on log
* Set a hard limit on the warning set to keep it from growing unnecessarily large
* Enforce a hard limit and improve message if full
* Enforce limit in log message because of possible threading adding past max
* Cleanup
* Log security mode change once per universe addition process
* Limit max size of the warning queue to 10
* Only allow to emit once per backtest
* Set limit as var, if at limit suggest more
* Always suggest more warnings may exist because we opted to log only once
* Improves greeks configurability and defaults for all option asset types
* Makes `StandardDeviationOfReturns` configurable by users, so that
greeks can be loaded according to user expectations and the series
of returns that they'd like to compute for `n` periods and timespan
of `T`, as well as resolution of the data in live mode.
* Changes resolution to max resolution available for the default
volatility model created for the security. Usually this only applies
to live mode, but if creating an instance of the
`StandardDeviationOfReturns` volatility model and no `updateFrequency`
is provided, the resolution's time span will be used as the default
value. Backwards compatibility for equities is maintained.
* Changes defaults for `StandardDeviationOfReturnsVolatilityModel`
to warmup greeks faster for other derivative asset types
* Improves comments on `StandardDeviationOfReturns` for clarity on how
to use the volatility model for end users
* Fixes bug where TradeBar could not have proper Symbol set when getting
max resolution
* Applies to QCAlgorithm.Universe and StandardDeviationOfReturnsVolatilityModel
* Adds tests to check volatility model is updated at specified config intervals
* Address review: add shared method for (Relative)StandardDeviation
volatility models
* Adjusts logic to determine bar type
* Address review: order by TickType when getting configs inside volatility models
* Allow USDC in cashbook without USDC-USD pair
* Cover more stablecoin cases unique to our crypto brokers
* Add unit test
* Cleanup and expand test cases
* Add USDCEUR and USDCGBP to GDAX Symbols
* Add missing tickers to SPDB
* Cleanup test
- Testing net5 uncovered these algorithms to be undeterministic
- Adjusting AllShortableSymbolsCoarseSelectionRegressionAlgorithm
internal implementation
- Order removal of universe members will be deterministic, when the
entire universe is removed.
* Fixes Double to Decimal Cast in GetAnnualPerformance
`GetAnnualPerformance` raises an exception if the `AnnualPerformance` calculation returns a double that cannot be cast to decimal (smaller than `decimal.MinValue` or bigger than `decimal.MaxValue`).
See `ProbabilisticSharpeRatio` where the same solution was applied.
* Updates SPY Market Data
SPY is a key asset since it is the default benchmark, and any change can lead to different `Alpha` and `Beta`
* Updates Unit Tests to Reflect Data Update
* Updates Regression Tests to Reflect Data Update I
Most of the regression tests change because of updated data (market and factors) of SPY (default benchmark) while the total trade remain the same.
* Updates Regression Tests to Reflect Data Update II
The following regression tests were changed to adapt to adjusted prices and keep the total trades:
- `BacktestingBrokerageRegressionAlgorithm`
- `LimitIfTouchedRegressionAlgorithm`
- `PortfolioRebalanceOnCustomFuncRegressionAlgorithm`
- `SetAccountCurrencySecurityMarginModelRegressionAlgorithm`
- `StopLossOnOrderEventRegressionAlgorithm`
- `TimeInForceAlgorithm`
The following regression tests have more trades since adjusted prices allowed more 1-2 shares trades that were rounded down to zero before:
- `FreePortfolioValueRegressionAlgorithm` 2 -> 3
- `PortfolioRebalanceOnDateRulesRegressionAlgorithm` 291 -> 298
- `TrailingStopRiskFrameworkAlgorithm` 5 -> 7
Especial cases:
- `AutoRegressiveIntegratedMovingAverageRegressionAlgorithm` 65 -> 52
- ARIMA model sensibility
- `BlackLittermanPortfolioOptimizationFrameworkAlgorithm` 18 -> 19
- BLM model sensibility
- `ExtendedMarketHoursHistoryRegressionAlgorithm` 20 -> 18
- Less minute bars before market opens
* Addresses Peer-Review
Fix `BacktestingBrokerageRegressionAlgorithm` to use `CalculateOrderQuantity` and round down `quantity` to an even number to pass a value assertion and update the expected value from 50 to 52.
The quantity calculated by `CalculateOrderQuantity` has changed from 50 to 53 because of factor file update.
* Add futures regression reproducing the issue
* Cleanup futures regression
* Add Options regression
* Add IRegressionAlgorithmDefinition
* Add IRegressionAlgorithmDefinition
* Adjust regressions to compare to expiration date; delisting time is not always correct
* Refactor subscription enumerator filtering of AuxData to top of the stack
* Update broken tests to reflect the split/dividends/delisting subscriptions
* Update regression statistics because of Delisting EOD instead of at time
* Let custom data configurations bypass filter
* Adjust expected count, now that we are letting aux data through to history requests
* another small adjustment
* Use ExpectedBarCount in error message
* Add filter test for both cases
* Add some clarifying comments
* Verify we did recieve data in the regressions
* Make _shouldEmitAuxiliaryData private readonly
* Filter out aux data for history requests
* Remove option to not include aux data in subscriptions
* Refactor filtering to be more explicit for each piece of data; fixes universe selection aux data
* Cleanup comments after removed var
* Refactor order of filtering for performance reasons.
`GetAnnualPerformance` raises an exception if the `AnnualPerformance` calculation returns a double that cannot be cast to decimal (smaller than `decimal.MinValue` or bigger than `decimal.MaxValue`).
See `ProbabilisticSharpeRatio` where the same solution was applied.
* Add SI
* Add ASI
* Add Swing Index
* Add Accumulative Swing Index
* Fix XML comment
* Add SI and ASI
* Add test data
* Add SI tests
* Add ASI tests
* Convert get only properties to methods
* Fix indicator name
* Replace special characters
* Fix indicator formula
* Replace test data
* Replace test data
* Update QCAlgorithm.Indicators.cs
* Minor format tweaks
Co-authored-by: Jared <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Fixed GDAXBrokerage missing fills caused by an incorrect assumption of a global counter for order fills (it is actually per-symbol)
- Filtered out rate limit messages in the fill monitor task
* Remove and replace OnEndOfDay() ref
* Restore functionality of obsolete EOD, waiting for deprecation in August 2021
* Cleanup
* Adjustments to only post message when using obsolete EOD
* nit, extra space
* Address review
* Adjust test to reflect new behaviour
* Move GetPythonArgCount to an extension method
* Add unit test
* nit accidental import
* Refactor broken test
* Use Py.GIL() state for extension
* Set _lastEmit before emitting, otherwise _workingBar is always null
* Aggregate bars if the data endTime is past lastEmit
* Add unit test
* Address Review
* Clean up unit tests
* Refactor solution to set consistent _lastEmit behaviour
* Add another unit test
* Make fixture non-parallelizable
* Undo last change, and adjust breaking test directly
* Reconcile duplicated code
* Add License header
* CS0219 Fixes: Value assigned, but never used
* CA1507: Use nameof in place of string literals
* CS0108 : Hides Inherited Member; Use new keyword to overwrite formally
* CS0114: Hides inherited member; use override keyword
* CS0168: Variable is declared but never used
* Tests CS1062; using obsolete implicit Symbol -> String; fix via .ToString()
* CS0472: Non Nullable Obj getting Null Checked
* CS0067 Member not used; ignore all cases for future use
* CS00162 : Unreachable code; either removed or ignored for debugging and test cases
* CS0169 Remove non-used fields; ignore those that may be used in future
* CS0414; Field is assigned but never used.
* CS0618; Obsolete properties and members; Only fixes simple ones, rest will have to broken up
* CS0649; Field never assigned too
* CS0659 & CS0661 ; Overwrite operators and equals but not hashcode; I don't really override it but just call base
* Small comment fix
* Cleanup pragma statement
* Tradier brokerage updates
- Add missing status check after REST API calls
- Initialize DataQueueHandler on-demand (on first subscribe call)
* Trigger build
`LiveOptionChainProvider.FindOptionContracts` handles the following data format:
`SPY 2021 03 26 190 000 C P 0 612 360000000`
where both existing `OptionRight.Call` and `OptionRight.Put` are declared in the same line.
* Remove regression references to non-existant Python versions
* Adjust regressions estimated capacity not adjusted by #5389
* Adjusts regression algorithms so that they pass (Index/Index Options)
* Changes start/end date on BasicTemplateIndexAlgorithm
* Changes option pricing model to BlackScholes in
IndexOptionCallITMGreeksExpiryRegressionAlgorithm
- The root cause of why there are no greeks at times for these
options was identified. It is most likely due to the underlying's
VolatilityModel not having had enough data to be "warmed up",
which means it will return a standard deviation of zero to the
option pricing model, rendering most metrics as NaN.
* Adds missing index/index options regression algorithms
- Regression algorithms are now 1-1 between C# and Python for
Indexes/Index options. All regression tests are now passing
* Fixes broken BasicTemplateIndex regression algorithm
* Previously traded SPY, but because we have no SPY data in Lean
master, I instead opted for index options, since data for those
dates is already included
* Deal with weekend for breaking test case
* Adjust DefaultEndDate test to always pass
* Check todays date for open
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
* Add support for Index SecurityType 🚀 (#5364)
* Add Index SecurityType 🚀
* Extend SecurityIdentifier & Lean Data classes with Index support
* Add Index SecurityType 🚀
* Extend SecurityIdentifier & Lean Data classes with Index support
* Fixes
* Added index cross basic template demonstration
* WIP: Prototype index security type for LEAN as non tradable asset
* Re-adds Index entries to MHDB after rebase
* First steps to getting Index Options running
* Looks at any instance where we pattern match for an option type
and replaces it with a generic call to `.IsOption()` for easier
extensibility in the future for additional option security types
* Adds IndexOption security and misc. classes
* Misc. changes, mainly related to any sort of special casing of
equity options and made index options take the same path
* Enables index options data for backtesting
* Adds new index options market hours to MHDB
* Misc. bug fixes for index options
* WIP: add live support for index options and indexes
* Use OptionMarginModel for Index Options because they both use the
same calculation for margin requirements
* Fixes contract not found errors on SPX index options and SPX index in IB
* Turns out index options' last trading day is the day before expiry,
which IB was expecting the last trading day.
* Add index option test cases (temp)
* LiveOptionChainProvider fix, use Symbol vs. ticker
* Description updates to regression algorithms
* Fixes bug in live trading for indexes and index options
* Adds overridable minimum price variation symbol property
* Adds variable sized minimum price variation for index options
* Adjusts symbol properties for index options
* Misc. bug fixes
* Fixes option assignment simulation for European options
* Updates index options regression algorithms (WIP)
* Fixes bug where index option exercise would trade index underlying
* Fixes bugs where SecurityType.Index was getting flagged as tradable
* Regression algorithms updates and addresses review
* Misc. style fixes and refactoring + a few bug fixes
* Updates regression algorithms to run without runtime errors
* Adds data for regression algos
* Sets DefaultOptionStyle on Canonical and support index options
* Update regression algos statistics
* Removes bad line in regression algorithm causing build to fail
* Minor tweaks
* Address review add comment about quoteBar parse scale
Co-authored-by: Balamurali Pandranki <balamurali@live.com>
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* RollingSharpeRatio Submission
Creation of RollingSharpeRatio indicator, utilizing LEAN engine. Tests were created and successfully passed by the indicator, using spy_rsr.txt as outside data file which was created during a prior running of the program.
* Fixed Test Case Loading and Added Indicator Function
Added the RSR() function to auto load the indicator in Algorithm\QCAlgorithm.Indicators.cs. Also added a reference for the test file ("spy_rsr.txt") in Tests\QuantConnect.Tests.csproj.
* Fixed SharpeRatio Indicator
Updated the base formula, indicator structure/call, and the testing data for the SharpeRatio indicator.
* Minor Fixes
- Removed dividend yield from calculation
- Fixed typos in code/documentation
* Minor Style Fix and Error Confirmation
- Changed style to match style guide better.
- Checking for confirmation that QuantConnect.Tests.Engine.DataFeeds.LinveTradingDataFeedTests.DelistedEventEmmited_Equity() is server side, as it runs locally and should not have been affected by prior changes
* Simplification and fixing of SR calculation
- Updates calculation to proper SR formula
- Utilizes IndicatorExtensions for SR calculation rather than manual calculation
- Defines counter for 1 extra period offset on warmup (otherwise first datapoint is inaccurate if warmup period is SharpePeriod length)
* Minor Fixes
- Removal of SharpePeriod and RiskFreeRate variables
- Substitution of (SharpePeriod + 1) for WarmUpPeriod
- Outdated comment fixes
* Resubmission for Mono Confirmation Error
Error detected on Git for this version of the program. An error should not occur as only comments and minor variables were edited. Resubmitting to see if it is a false positive mono error.
* Notation Fixes
Fixed notation regarding spacing and outdated comments.
* Removal of Counter Logic
- Replaces counter for warmup with improved IsReady logic
* Fixed Syntax
- Removed unused imports
- Fixed variable readability
- Fixed comments
- Replaced Tabs with spacing
Co-authored-by: quantify-cflynn <quantify-cflynn>
* Update to new QC PythonNet
* Update readme
* Remove Python.Runtime config, replaced by env var
* Allow local packages testing in repo
* Address Review
* Add the LocalPackages readme
* Update Jupyter Image
* Update Research ReadMe
* Adds backwards compatibility for MHDB and SPDB for new SecurityTypes
* Skip invalid SecurityTypes in MHDB and SPDB
* One time log invalid SecurityTypes in MHDB/SPDB
* Moves logging of SecurityType into new extension method
* Adds TryParseSecurityType to Extensions
* Adds CapacityEstimate and SymbolCapacity
The capacity estimation has been moved from
the report generator and wired directly into
Lean via the ResultHandler. In addition,
the capacity estimation strategy has changed
to account for errors in the previous iteration
of the capacity estimation.
Many many thanks to Jared for being much of the
mastermind behind this project. It would have
been harder to complete without him to bounce ideas
off of.
* Moves old tests to regression algorithms
* Adds Estimated Capacity statistic
* Removes old capacity estimation tests
Final report capacity estimation. Pushing to save state
* Fixes bugs, cleans up code and adds comments
* Adds forced sampling to Capacity Estimation
* Misc. bug fixes for daily data
* Updates capacity test cases' Estimated Strategy Capacity statistic
* Adds Capacity Estimate to all regression algorithms
* Removes Report's StrategyCapacity class and fixes bug in tests
* Adds null check in BacktestingResultHandler to fix
BacktestingTransactionHandler failing tests
* Deletes old capacity estimation classes
* Retrieve capacity estimates from backtest statistics results
instead of calculating at runtime
* Make $0.00 capacity return as "-" and Result = 0 in report
* Adds capacity to runtime statistics
* Converts capacity to number denoted by financial figures in RuntimeStats
* Addresses review: code cleanup for Capacity and adds comments to regression tests
* CoinApi HistoryProvider impl.
* Small fix of ToTradeBar()
* Revert the change : CreateSliceEnumerableFromSubscriptions
* More tests and fixes.
* Removes redundant ToList()
* too much spacing
* Fixes to get data for an unlimited period of time, by means of pagination
* Add options support
- Added IDataQueueUniverseProvider implementation
- Fix subscribe ticker for options
* Tradier DataQueueHandler web sockets implementation
- Equities and Options (trades and quotes)
* Add error handling
* Add support for options
* Fix log messages
* Address review
- Updated Tick constructor usage
- Use $empty$ symbol for last unsubscribe message
- Added symbol mapper unit tests
* Enforce USD account currency if allocation is limited
- Enforce USD account currency if allocation is limited. Adding unit
test
* Adjust currency log message
* Reduce live coarse selection interval check to 10 min
- Reduce live coarse selection interval check to 10 min. Adding unit
tests, reducing code duplication in tests.
* Adjust unit test after self review
* Removes SevenZipSharp library from ToolBox
- Library removal required for update to .NET 5.0 since it's not
compatible with it and Linux is unsupported.
* Adds new extract 7z functionality to Compression project
* Refactors AlgoSeekFuturesConverter 7z extract
* Refactors DukascopyDataDownloader 7z extract
* Addresses review: code cleanup + exception type change on timeout
* Removes not needed stream of data in Dukascopy downloader
* Makes output directory non-optional
* Fixes potential mapping issue, caused by not mapping Symbols before
reading data off disk
* Fixes issue where some results would evaluate to zero capacity
* Loads of refactoring, mostly resulting in cleaner code
- After https://github.com/QuantConnect/Lean/pull/5207 history requests
are selecting the correct data type and cause the regression test to
fail because there is no QuoteBars for daily equity. Updating and
fixing this.
- Centralizing the logic around getting the SubscriptionDataConfig to
use
* Filter values that are before subscription start time; also adjust starttime for OpenInterest
* Use data EndTime for comparison
* Allow Auxiliary data through
* Fix OpenInterest DataReader Logic
* Add regression
* Address review
* Ignore open interest for time slice
- TimeSliceFactory will directly ignore open interest for determining if
the slice has data or not. Open interest will still be available
through the Tick collection. Reverting some of the previous commits
changes since they are no longer required.
- HistoryRequests and SubscriptionRequest will use AlwaysOpen exchange
for open interest requests. Adding unit test reproducing issue
- Adding `BaseDataRequest` to avoid duplication logic.
* Make OpenInterest an internal feed and ignored by default in history
- Adding unit tests
* Revert SubscriptionFilterEnumerator Start time addition
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Adds Static AlgorithmCreationTimeout Property to BaseSetupHandler
This value will be used to set maximum time that the creation of an algorithm can take.
* Adds Additional Logging to AlgorithmPythonWrapper
It will informing the user how long it takes to import the module.
* Changes How AlgorithmCreationTimeout is Initialized
It will prevent callling Config.GetDouble on every call.
* Add support for live trading delisting events
- Adding `LiveDelistingEventProviderEnumerator` that will emit delisting
events. Adding unit tests
- Remove unused `LiveAuxiliaryDataEnumerator`.
- Refactor the corporate event providers being used per security type
* Self review: compare date component
* Add new LiveDataBaseDelistingEventProvider for equities
- Add support for delisting events for equities. Adding unit tests
* Fix cleanup of delisting equity subscription
* Refactor IPrimaryExchangeProvider.GetPrimaryExchange
now it returns a PrimaryExchange instead of a string
* Update MapFilePrimaryExchangeProvider.cs
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* To maintain consistency between calculations, we will use
the end of day equity value to calculate the returns per day.
This fixes a bug where daily equity series would zero out and result
in an exception being thrown since no elements were being passed to
the Sharpe calculation.
* Implements wrapper to get the exchange as a single letter representation
Also replace lower 'l' with capital 'L' in condition flags enums. it is a new warning on the compiler and it looks cool :)
* There are cases where the exchange is null or empty, in those cases return null
* Add tests for single character exchange representation
* Update GlobalTest.cs
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Calculate PSR for live algorithms in report generator
* Use live series for calculating of CAGR, Max DD, and PSR
* Removes Kelly Estimate from report and template.html
* Update template.html
* Rolling Beta bug fix when using daily data
* Max drawdown output as percent
* Make days live report element use equity curve last day instead of today
Co-authored-by: Jared <jaredbroad@gmail.com>
* Adds LimitIfTouched order.
TODO:
-- Add tests.
-- Add into existing regression algorithms.
-- Refactors (?)
* Fixes
- Remove unrequired space changes
- Fix EquityFillModel min/max limit price fill
- Add TriggerPrice for UpdateOrderRequest.
Quote info used w.r.t. comparing against Limit prices for LIT
FillModel.cs implementation is fixed to use quotes when comparing against set limit prices.
Also changes test implementations to assert which of quotes/trade-bars are being used
Reviewer-suggested fixes
------------
Merge remote-tracking branch 'origin/limiftouched' into limiftouched
Styling
Adds missing null check for quotebar
Styling
Adds missing null check for quotebar
Merge remote-tracking branch 'origin/limiftouched' into limiftouched
High/Low w.r.t. trigger price for determining if TriggerTouched changed to Current price
0462ad668a (r569832380)
fill fixes:
FillModel.cs: Fills exactly at the limit
EquityFillModel.cs: https://www1.interactivebrokers.com/en/index.php?f=608
Equity fill now uses quotebars
Revert - use new constructor which emulates former SubmitOrderRequest
Style changes
Reverts order model to original by means if new constructor
High/Low w.r.t. trigger price for determining if TriggerTouched changed to Current price
0462ad668a (r569832380)
fill fixes:
FillModel.cs: Fills exactly at the limit
EquityFillModel.cs: https://www1.interactivebrokers.com/en/index.php?f=608
Equity fill now uses quotebars
Revert - use new constructor which emulates former SubmitOrderRequest
Style changes
Reverts order model to original by means if new constructor
Merge commit 'bf4c96d2a055ea808fa4293662528c11a89b72c7'
* Suggested style fixes
* Review fixes
-------------------
- Suggested style fixes
- Changes LIT regression to better incorporate order modifications
- TODO: orderlisthash must be fixed
Fixed LIT regression algo
-----------------------
- Includes asserts in OnOrderEvent
* Fix OrderListHash
OrderListHash -> -292689487
* Re-adds quote nullchecks
* EquityFillModelTests fixes asserts
* Reordering FillModel.cs
* Fixes quote logic, adds methods in FillModel.cs
* Refactoring + LIT regression fixes
-- revert unneeded changes
* Fixes list hash
* Rebase -- catch up upstream
* OrderListHash fix
* Various fixes by reviewer
* Final requested changes
* tagged time -> utcinvariant
* Fixes listorderhash
* Time changed to UtcTime.ToString(DateFormat.US, CultureInfo.InvariantCulture)
* Adds Python LimitIfTouchedRegressionAlgorithm
* adds LimitIfTouchedRegressionAlgorithm.py
* adds LimitIfTouchedRegressionAlgorithm.py
* Minor changes to LIT regression algorithms
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* fixing up Coin API streaming
* Fixing file name
* Adding log showing CoinApi product plan
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* .NET Core/5.0 will not compile when calling
AppDomain.CurrentDomain.DefineDynamicAssembly() since it was removed
in .NET Core.
AssemblyBuilder replaces this specific functionality since
AppDomains were deprecated in .NET Core
* Update OrderListHash to use MD5 as hash instead of hash code
* Update regression algorithm OrderListHash statistic
* Use full MD5 hash as OrderListHash, update regression statistic
* Fixes failing regression tests
* Minor logging improvements
- Remove Thread.Sleep() call when logging debug
- FileLogHandler will capture lock after generating message to log
- ConsoleLogHandler will log time as UTC, cheaper than converting time
zones
* Fix typo
Fix typo
* Updates CPU performance metrics gathering for project modernization
* PerformanceCounters are not supported cross-platform in
.NET Core/5.0, which requires the use of an alternative method
in the gathering of performance metrics. Since no exposed .NET API
exists to gather these metrics without blocking, a new thread is
created to block the time necessary for the CPU performance to be
calculated.
* Address review: remove ResetEvent and make CpuPercent atomic
* Dispose of CpuPerformance instance before exiting Lean
* Minor tweak for Task and comment
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Some app.config files were kept to either enable features
that are only enable-able via the config files, or binding redirects
for DLLs that are not packaged with NuGet (i.e. VisualStudio
package, required to run tests successfully @ 15.0.0.0)
* fix
* Minor tweaks adding unit tests
* Address reviews
- BrokerageSetupHandler will use UniverseSettings.Resolution as default
resolution
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Atreyu Brokerage initial setup, and changes required for integration with main Atreyu project
* Address reviews
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
Bitfinex exchange may return us an empty result - if we request data for a small time interval during which no trades occurred - for example 1 minute interval - can happen even with most liquid pairs, like "ETHUSD" - would be good to have some time margin for such scenario. and pump more data to warm up conversion rates
* Cleans history for ArimaIndicator/TimeSeriesIndicator.
-- removes commits from a tracked, already merged branch
-- removes artifacts from debugging sessions
* Removes AR/MA method as a user-specifiable method.
-- Prevents need to reference dll for MathNet in Tests (and potentially elsewhere).
-- Wrapper can be implemented around this functionality.
* Removes AR/MA method as a user-specifiable method.
-- Prevents need to reference dll for MathNet in Tests (and potentially elsewhere).
-- Wrapper can be implemented around this functionality.
* Better adherence to established code style
* Makes _intercept = true by default in constructor where it is not parameter
* WIP -- addressing reviews
* Passing tests following prior refactor
* Rearranged code, access modifiers adjusted
* Fixed indexing of _mafits, adds example algorithm
* Adds regression algo in python + addresses some refactors
* Addresses review
* Adds regression stats
* Fixes missing value signs
* Removes redundant code
* style changes
* style changes
* style: "err" -> "error"
* Minor tweaks
* Fixes python arima regression test
* Refactors AutoregressiveIntegratedMovingAverageTests.cs
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Fix IB future chain being started at reset hours
- Fix IB future chain being started at reset hours. Adding unit test
* Refactor CanAdvanceTime into CanPerformSelection
* Migrate TradeConditionFlags and QuoteConditionFlags to Lean
* Address revirew
Implement flag systems using long, this allows us to declare up to 64 flags
* Updates NodaTime to version 3.0.5
* Updates code to ensure compatibility with new NodaTime version
* Updates System.Runtime.CompilerServices.Unsafe to 5.0.0
- Required by NodaTime
* Adds comments explaining changes in NodaTime
* Updates and adds new Python packages to DockerfileLeanFoundation
* Remove torch-geometric packages due to import issues
Torch geometric packages installed from pip as they are here result in an error saying to install the CUDA version of the packages. When the CUDA version is installed, we get a symbol error when attempting to link the `_version.so` shared object.
* Add notification targets and events to live packet
- Add notification targets and events to live packet. Adding unit tests
* Add project name to job packet
* Add ShortToString and WebNotification Header
- Adding ShortToString implemnetation of OrderEvent and Insight
- Add Web notification headers
* Add more unit tests
* Revert json lower case fields
* Include MortgageRateVolatilityAlpha and fixes
* Include PlaceHolder PythonAlgorithm.cs
* Delete BinanceUtil
* Remove unused Regression Algorithms
* Minor tweaks
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Upgrade IBAutomater to v1.0.40
* Fix FinancialAdvisor account id check
- The Ixxxxxx account code is the Master Account for Fully Disclosed Brokers and this account is not tradable. Proprietary accounts for Brokers/Dealers used for proprietary trading have the Uxxxxxx account code.
* Update IBAutomater to v1.0.42
* Update IBAutomater to v1.0.43
* Log server version in ConnectAck handler
* Adjust delisting liquidation time
- Adjust delisting liquidation time to 15 min before market closes.
Adding unit tests. Updating existing.
- Handle `Statistics.CompoundingAnnualPerformance` invalid calculation
to avoid exception.
- AlgorithmManager will not handle delisting events in live trading
- Fix bug where due to a split driven liquidation matching delisting
date a position in the option would remain open. Reproduced by
`BasicTemplateOptionsFrameworkAlgorithm`
* Address review
- Address review add documentation on delisting offset span
* Fix Symbol Alias
- Fix Symbol Alias being wrong in some cases for futures and options
* Add test cases
* Address review
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Add main exchange to map file
minimum minimorum test
* Add MainExchange to MapFilRow IEquatable implementation
Also write CSV line correctly
* Normalize exchanges
Also add small test form MapFileRow
* Exchange name normalization
- Exchange name normalization.
- Tick will protobuf `ExchangeCode` which will set existing `Exchange`
* Include single character exchange code
* Encode exchange as byte, Include exchange name as Enum
* Update MapFileRow.cs to accept a byte instead a string for primary exchange
* Fix issues in encoding exchanges map
* Fix broken MapFile tests
* Rename Main Exchange property in MapFileRow
* Use BATS instead of BATSZ
* Add primary exchange from map file
* Move MapFilePrimaryExchangeProvider to its own file
* Implement Char for primary exchange in map files
* Fix broken tests after implementing char exchange into map file
* Add a new constructor for MapFileRow
it accepts a PrimaryExxchange object as parameter
* Fix broken test
WIP
* Address review
WIP
* Handles unexpected encoded exchanges
* Keep addressing review
* Address review
* Fix broken tests
Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated
* Revert breaking change.
* Add a check that resolved map file actually have content
The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm.
The resolver returns a not-null MapFile, but it is empty.
To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV.
* Set PrimaryExchange.UNKNOWN as default
* Address review couple minor bugs
- Use SID market instead of USA
- Default ticke exchange is UNKNOWN
* Update map files with latest format and latest info
This include:
- VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX
- AAA delisting and a new listing with ticker AAA
- FOXA ticker changue to TFCFA and later it was delisted
* Update test after updating latest map files
* Fix bug with factor files after updating map files
* Fix bug with factor files after updating map files
and revert statistics in regression tests
* Update OrderHash in regression test
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Have BrokerageModel determine default benchmark
* Add DefaultBenchmark to Python wrapper
* Handle Null benchmark case
* Add NullBenchmarkRegressionAlgorithm
* Refactor solution to have BrokerageModel return IBenchmark; also refactor QCAlgorithm benchmark handling
* Always create a new security for benchmark
* Drop security overload, Always create a new security for benchmark
* Check our securities for a symbol matching the ticker before creating a new one
* No Python version of this regression
* Address review
* Create shared SecurityBenchmark creator function
* Add Python regression and needed FuncBenchmark constructor
* Fix delisted liquidation orders being cancelled
- Place delisted liquidation orders 10 min before market closes of 10
min before the end of the delisting warning date. Adding regression
test and unit tests. Updating existing tests.
* Fix failing python option unit tests
* Fix bug where positions would be open delisting liquidation
* Fix universe selection and delisting
- Delisting will happen ASAP for all types. Giving priority to close
positions on derivates first
- Fix bug in universe selection where OptionChain would remove
underlying even if holding a position in derivate.
- Updating regression tests statistics
* Add unit test, fix unit test expected stats
* Fixes ETB/shortable regression algorithms that were previously failing
* Addresses review and fixes bug
* Fixes bug where orders would be denied regardless of direction
whenever they exceeded the absolute value of the shortable quantity
* Updates regression algorithm + statistics + simplifies test
* Fixes python regression algorithm
* Addresses review: removes comments and unneeded imports
* Added missing early closes for 1999-2008
See GitHub issue for details
* Fix expected Treynor ratio for backtest in test suite
This backtest runs from 2008-10-10 to 2010-10-10 so it is affected by the early close date now recorded on 2008-12-24.
* Creates IShortableProvider interface + interface impls
* This is the foundational work for the addition of the shortable stocks
feature for backtesting and live trading. Note that the QCAlgorithm
API and the backend transaction handling will be implemented
separately.
* temp; work on preorder checks
* improve checks
* tmep
* Enforces ETB checks at BrokerageTransactionHandler
* Adds ETB Shortable Provider to QCAlgorithm and IAlgorithm
* Removes ETB check from PreOrderChecksImpl
* Removes outdated test, new test to come soon for relevant class
* Work in progress commit, pushing for review.
* Adds new regression algorithm (WIP)
* Enhances ShortableProviderOrdersRejectedRegressionAlgorithm
* Adds new methods to QCAlgorithm
* Move IShortableProvider to have BrokerageModel own it
* Comments updates
* Adds new properties to Equity Security for shortable
* Fixes bug where retrieving open order quantities would aggregate the
submitted order quantity rather than the remaining order quantity for
open orders.
* Fixes bug where quantity of zero would result in a
false positive in QCAlgorithm.Shortable(...)
* Code refactoring and comments updates
Co-authored-by: Jared <jaredbroad@gmail.com>
* Migrate to newer CSProj format; build succesful
Rebase onto master
* Maintain binaries directory
* Reinstate FSharp default condition
* Remove reference to packages.config
* Test project fixes
* Remove nuget.config and "Solution Items"
* Remove repetitive assembly info
* Reinstate default compiles and cleanup redundant settings
* Reinstate any specific settings that defaults are unclear for
* Reinstate Framework.nuspec
* Make test internal classes public for composer use; undo assembly change
Rebase onto master part 2
* Address review; restore some settings
* Restore RootNamespace and AssemblyName attributes
* Use NugetPackageRoot instead of NugetPackageFolders
* Reinstate "RestorePackages" setting
* Transfer compilied Python files to launcher and tests bin
* Remove conflicting settings
* Fix WebSocketSharpFork references
* Fix FSharp Core reference
* Upgrade TestPlatform.ObjectModel to 16.8.3, and redirect V11 binding
* Add Mono.Cecil package
* Address review
* Add interface to allow custom security entries for MHDB and SPD
* Simplify adding custom Properties and MarketHours via AddData overload
* Refactor
* Remove AddData and GetDatabaseSymbolKey overloads
* Add unit tests
* Remove AddData overload, for real.
* Nit changes
* Set CustomDataBitcoinAlgorithm back to original
* Add Python and C# Regression
* nit typo
* nit typo actual
* Reset symbol property to default
* Reflect last change to py regression
* Revert "Remove AddData overload, for real."
This reverts commit dc877495c0.
* Implement AddData solution fully
* Function nit fix
* Address review
* nit - remove param comment
* Address review
* Adds FuturesOptionsUnderlyingMapper
This adds a mapper for Futures options that have a different
contract month between the FOP and the underlying future, as is
observed in the contracts ZB, ZC, ZS, ZT, ZW, HG, SI, GC.
An example of this is GC (e.g. OGH21 -> GCJ21). Other contracts follow
different rules, which have been included as part of this commit.
* Adds unit tests for FuturesOptionsUnderlyingMapper
* Adds unit and regresssion tests for underlying Futures mapping for FOPs
* Addresses review and adds FuturesListings
* Removes dependency on IFuturesChainProvider in
FuturesOptionsUnderlyingMapper
* Fixes bugs related to FuturesOptionsUnderlyingMapper and incorrect
results
* Modifies misc. code to handle new results correctly from FOPs
underlying mapper
* Makes FOPs underlying mapper static, and makes other methods private
* Adds new tests for FuturesListings
* Addresses review: code cleanup
* Address review: Makes arrays to List in FuturesListing private methods
* fill forward data points until expiry date
* wip: test non Delisting
* handle Aux not-Delisting dp; keep order while ff'ing
* simplify test checks
* wip
* fix liquidation expected date
* change statistics
* check EndTime, because it's used for time synchronization
* clean up code; remove duplicated test
* Fix MACD WarmUpPeriod and Updating
* Add System to use Math library
* Fix WarmUpPeriod math and add tests
* Fix SchaffTrendCycle Indicator WarmUpPeriod
* Ensure fastPeriod < slowPeriod
* Minor tweaks
- Remove unrequired Math.Max operation
- Remove unrequired changes in solution file
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Update RandomDataGeneratorProgram.cs
Replaced AddMonths(6) with a Datetime value half way between settings.Start and settings.End.
* Update RandomDataGeneratorProgram.cs
* Update to bug-5030-CFDdatanotoworking
* Added midpoint unit test
* Minor test assert improvement
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* LeanDataReader future option data
- The LeanDataReader will be able to read an entire zip of future and
option data, currently it's just returning the first entry in the zip
file. Adding unit tests
* Fix added unit test data path
* Universe dispose and removal
- Fix bug where in some cases the disposed universe would not get removed
from UniverseManager not allowing new universes to get added correctly,
using the same symbol. Adding regression test
- Fix bug where in some cases selected symbols would not get removed
when the parent universe would get disposed of.
- Minor normalization on UniverseSelection call
* Address review
- Add status plots for the added regression test
* Adding regression algorithm reproducing issue 3914
* change filename template. add fillforward resolution suffix
* replicate GH issue 5116 on master
it's easy to reproduce on FXCM market by using data resolution different from fillforward resolution.
in this cases daily vs hour/minute were added
* change priorities of Time & EndTime values.
it's necessary to calculate EndTime properly, and then we can align Time to it.
potential end time should be also calculated using ptoper TZ. Because of we store open hours without TZ movement TZ in market-hours it's necessary to reapply TZ
fix tests
* Miss 2AM bar on Sunday of DST; FF 2AM bar on Sunday ST
* use UTC TimeZone as baseline during comparison
* Refactors Market Fill Model
Create `GetBidPrice` and `GetAskPrice` methods to request the most suitable price to fill market orders.
Fix unit tests to show that new implementation prevents using non-market data to fill the order.
* Addresses Peer Review …
Modifies EquityTickQuoteAdjustedModeRegressionAlgorithm to reflect changes in the market fill method. The previous implementation only looks for the last tick when it should look for the last tick of quote type, so current implementation is an improvement.
* Fixes for IB malformed symbols
- Add handling for IB future malformed symbols. Adding unit test
- Fix IB option malformed symbols which had more whitespaces than
expected. Adding unit test
* Address review
* Adds data parsing for malformed option contracts in IBBrokerage
* Some contracts come in malformed from IB when downloading our
account holdings. We attempt to detect whenever this happens with
our options Symbols and try to recover the contract.
* Addresses review and fixes bug
* Uses contract currency for the new contract created from the
malformed contract.
* Bug fix: Sets ibSymbol to new Symbol of the contract once the
malformed contract has been parsed.
* Adds support for Futures and FOPs
* Refactoring code and cleanup
* Addresses review: code cleanup and refactoring
* Addresses review: remove redundant logging of contract in log statement
* Silence DotNet in building stubs
* Reapply silence to stub publishing
* Have QuantBook store and restore initial LogHandler
* Add assembly action to maintain LogHandler
* Allow AlgorithmRunner to use ConsoleLogHandler
* Use MaintainLogHandlerAttribute to provide current test logger
* Refactor LogHandler creation
* The test didn't take into account that the contract expiring on
December would mean that there would temporarily be no Dec. FOP
contract. We fix this by looking for the Mar. contract instead
if the december contract has expired.
* added test covering minValue / maxValue issue with JsonRoundingConverter
* change namespaces
* JsonRoundingConverter fix decimal.Min and MaxValues (cannot deserialize)
* remove dependency on 3rd party library
* c# 6 compatible code (remove pattern matching)
* Fixes BacktestResultPacket deserializing
- Serializing decimals as strings to avoid precision loss, since json
convert will use floating point precision. Updating unit tests.
- Fix logging unit test failing to delete file being used.
Co-authored-by: Mark Virchenko <mark.virchenko@calienteam.com>
* test
* wip
* Revert "Fix duplicated history entries when contains daylight saving time change (#4700)"
Use proper rounding down
* regression test
* remove unused parameters
* more tests
* fix name and comment
* improve regression test
* more tests: oanda market hours
* re-apply Exchange TZ to bar EndTime
* fix expected results
* we can't substract minute because it can harm algorithm on minute resolution; so we could use tick?
* rename prop: conflict with QCAlgorithm.StartDate
* do not log messages to pass travis ci log limit
* assign loghandler in AlgorithmSetupHandler
* reference to PR for more description
* due to https://github.com/QuantConnect/Lean/pull/5039 we don't need to override it manually
* Adding Api.ReadBacktest optional getCharts
- Optionally allow users not to fetch backtest charts when reading a
backtest, it can be slow
* Fix null reference for deleted/cancelled backtests
* Get Leaky bucket config settings once
* Reduce Travis setup verbosity
* Introduce ConsoleErrorLogHandler
* Change Console.WriteLine to Log statements
* Quiet wget
* Route build stubs stdout to null
* Fix Quantbook history test
* Silence stub packages directly
* Use parameterized log-handler for testing
* Rename AssemblyInitialize Setup
* Fix AlgorithmRunner file logging
* Drop all overriden LogHandlers in tests
* Change to OneTimeSetup to maintain LogHandlers
* Permit any ILogHandler to be defined in params
* Fix for AlgorithmRunner Handlers V2
This layer is designated for common settings that enforce your team preferences for the current solution.
Since we don't have specific resharper settings we want to share within the team we ignore this file
* Adds DeM indicator
* Added reference to param movingaverage type
* Fixed variable declarations
* Added nameless initialize
* Missing DeM "type" args added
* Missing DeM "type" args added
* refactor
* Undid _previousInput → protected
* Demarker symbol: DeM →DEM
* Symbol change: DeM → DEM
* Updated symbols
TestDivByZero originally had dem as cmf.
* Symbol: DeM →DEM
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
- Adding optimization Id to backtests packets
- SeriesSampler will allow truncating the samples
- Removing OptimizationEstimate, simplifying getting estimate and
runtime stats separatly
* Fixes intraday delistings not occurring for Futures and FOPs
* Previously, we would wait until the next market open to
liquidate futures and futures options contracts. Since these
contracts can not be traded at the next market open and require
intraday delisting, changes were made to liquidate at the first
available place where we know the market is open. This means
we now liquidate futures and FOPs intraday as a market order.
* Maintains backwards compatability with equities and equity options
delisting behavior
* Addresses review: adds additional protections for ProcessDelistedSymbols
* We choose to adjust the delisting date to the next market open only
if the market is not open at the current time, otherwise the time
would have been adjusted to the market open of the next trading day
* Addresses review: reverts changes and fixes error message in regression algo
* Adds Futures Options History Research Support
* Address review: make canonical future throw when calling GetOptionHistory
* Improve error message, recommending users to user FutureChainProvider
* Adds the awesome oscillator.
* added missing type hint for AO
* cleaned initializations
* refactor in call for AO(fast,slow,type)
* added missing type parameter for AO
* Changes AO sub-indicators to public.
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
* Fixed Python runtime issue that was occurring when trying to generate reports locally on OSX/mono, but assume the issue impacts all configurations.
* Move Python.Runtime config to common
* Remove duplicate files
* Update readme
* Typo
* Change destination in build directory
Co-authored-by: Charles Naccio <cnaccio@gmail.com>
* Added CMF indicator
CMF is a volume-weighted average of accumulation and distribution over a period.
* Added initializer for CMF
Registration for ChaikinMoneyFlow implemented.
* Added CMF tests.
* Added CMF tests.
* spy_cmf.txt changed to external indicator data.
* Implement suggestions of @AlexCatarino
* added sum terms as subindicators.
* added sum terms as subindicators.
* Removal of vestigial rolling window
* Minor nit changes
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* WIP Backtest adjustments
* Workaround awaiting changes
* Adjustments to API Changes
* Nit fix
* Custom Newtonsoft Deserializer for AlphaRuntimeStatistics
* Workaround Travis build error
* Drop AlphaRuntimeStatistics converter; use Decimal converter
* Use StringDecimalJsonConverter
* Undo set properties
* Add more members to de-serializer class
* Adds preliminary universe selection for Future Options
* Fixes scaling issues with Future Options
* Fixes scaling multiplying by 10000x instead of using _scaleFactor
* Fixes scaling for Tick
* Revert changes to Tick since it divides the scaling factor
* Changes stale method name to new method name after rebase
* Fixes selection bugs, adds new methods, and adds unit tests
* Fixes bug where Equity Symbol was created for an underlying
non-equity Symbol, resulting in equity data trying to be loaded
* Adds unit tests covering changes to Tick, QuoteBar, TradeBar and
LeanData
* Adds regression test for AddUniverseOption filter contract selection
for Future Options
* Addresses review - modifies the AddFutureOption signature
* Adds new AddUniverseOptions method overload
* Removes and adds a new unit test
* Misc. modifications to account for new changes
* Fixes bug where futures were loaded using default SID Date
* Refactors and removes unnecessary work
* Fixes regression algorithm, which previously made no trades
* Adds future option data
* Adds the corresponding underlying data, in this case, futures data
to enable usage of future options data
* Replaces data with new data (ES18Z20)
* Improves Future chain filtering and updates regression stats
* Add AddFutureOptionContract API
* Expands regression and unit tests to test in finer detail
* Adds Python regression algorithms for AddFutureOption[Contract] methods
* Adds new unit test for BacktestingOptionChainProvider
* Fixes bug with BacktesingOptionChainProvider where we
attempted to load the Trades option chain first, resulting
in breakage of backwards compatibility and limitation of the
option chain.
* Adds new regression algorithms (Py) to Algorithm.Python project
* Adds FutureOptionMarginBuyingPowerModel
* Modifies code paths used to select margin model
* Adds related unit tests for margin model
* Fixes issue with unit test and MHDB/SPDB lookup for Future Options
* Preliminary regression algorithm testing ITM call/put option buying
* Fixes bug where fee model used did not find non-US market
options fee model. We now use the futures fee model for future
options because IB charges the same commissions per contract
between futures and futures options
* Adds proper regression algorithm for ITM future options expiration
* Pushing broken algorithm for review
* Currently, algorithm does not fill forward, causing
a single future option to not get exercised when it is delisted.
* Adds FutureOptionPutITMExpiryRegressionAlgorithm
* Improves existing regression algorithm for call side
* Fixes bug in existing regression algorithm
* Adds AAPL daily data to advance enumerator for ^^^ fix
* Adds additional future option regression algorithms
* Adds Buy OTM expiration regression algorithms
* Adds Sell ITM/OTM expiration regression algorithms
* Adds missing Python regression algorithms
* Adds remaining Python regression algorithms and fixes issues
* Fixes naming issues and statistics
* Adds short option OTM regression algorithms (Py)
* Add license header and class comments to python algorithms
* Cleans up comments and docstrings
* Create Buy/Sell call intraday regression algo
* Redirects future options symbol properties to futures symbol properties
* Asserts exercise/assignment price and updates stats in regression algos
* Adds new unit test covering changes to SecurityService
* Adds comments and fixes failing test
* Partially fixes future option mis-calculated profit/loss
* Adjusts portfolio model to calculate FOP as a no upfront pay asset class
* Updates regression algorithm statistics
* Begin IB FOP support
* Initial support for FOP IB data streaming, live í¾
* Adds additional functionality to LiveOptionChainProvider
- Allows querying CME API to retrieve option chains for CME products
- Ultimately, it's also the groundwork for the CME
LiveFutureChainProvider
* Edits IDataQueueUniverseProvider interface to provide greater
control to implementors of it
* Misc. bug fixes required to get FOP data streaming through IB
* Adds comments, adds missing rategate call, and cleans up code
* Force exchange for FOP and Futures when no exchange is provided
* Fixes bug with Portfolio modeling across all asset classes
* Adds LiveOptionChainProvider tests for Future Options
* IB brokerage option symbol bug fixes and improvements
* Fixes contract multiplier lookup bug
* Fixes issue where we attempted to subscribe to IB data feed with canonical security
* Adds ES MHDB entry
* Reverts portfolio modeling changes for Futures Options
* Since IB eats into our account's cash balance when
a new FOP contract is purchased, we must model by applying funds
to our cash whenever a new purchase/sell occurs.
If we choose to model FOPs exactly as we do with futures, we
will end up with an invalid TotalPortfolioValue on algorithm
restart. By all means and purposes, FOPs are modeled exactly
the same as equity options with respect to the portfolio.
* Adds comments clarifying portfolio modeling and clarifies
existing portfolio modeling comments with additional context.
* Fixes IB symbol lookup for future options
* Fixes LiveOptionChainProvider looping 5 times per option chain
request, even on success
* Sets OptionChainedUniverseSelectionModel to produce a canonical
future/future option/option Symbol to avoid creating two Symbols
* Adds GLOBEX future option symbol mapping from future -> fop
* Fixes LiveOptionChainProvider loading wrong contract option chains
* Fixes loading of futures options ZIP files when backtesting
* Adds a string -> decimal JSON converter
* Additional fixes/refactoring to the LiveOptionChainProvider
* Adds tests for changes to Symbol and LeanData
* Reverts changes to IB-symbol-map
* Fixes Value for mapped future options tickers
* Fixes Symbol test
* Changes path of future options to future's expiry date
* Extra changes made to remove scaling from writing CSV
* Added method to map from FOP Globex -> FUT Globex
* Fixes MOO and MOC orders for future options
* Note: this order type might not be supported by IB or CME.
* Bug fixes and updates unit tests
* Update regression tests and data format
* Rebase changes
* 1. Multiple bug fixes for LiveOptionChainProvider, reverts IQFeed changes
2. Address review (partial): Code reuse and cleanup
1.
* Modifies check in
`AddFutureOptionShort(Call|Put)ITMExpiryRegressionAlgorithm`
to ensure no buys have negative quantity
* Code reuse changes in IB brokerage
* Bug fix in IB brokerage where we assigned the FOP expiry
as the futures expiry (requires verification)
* Doc changes and adds missing summaries/license banners
* Disposes of HTTP client resources in LiveOptionChainProvider
* Renames classes and adds FutureOption folder in Common/Securities
2.
* We revert back to the quotes API for the option chain,
since the settlement API sometimes had missing strikes.
* Fixes future option expiry being set as future's expiry
in LiveOptionChainProvider
* Fixes bug where wrong option chain was selected because of bad
expiry lookup in the futures expiries returned from CME
* Fixes multiple looping bug in LiveOptionChainProvider
* Adds strike price scaling for LiveOptionChainProvider
* Reverts IQFeed changes and simplifies interface upgrade changes
Some additional challenges we'll have to solve as part of FOPs:
- The `OptionSymbol.IsStandard` method makes the assumption that
weeklies contracts follow the pattern equities follows, which
does not apply to Futures Options
- The Subscription created in:
`OptionChainUniverseSubscriptionEnumeratorFactory`
...adds a Trade config. For illiquid contracts, this
will delay universe selection for the option symbol
until we get a trade. However, if we add a quote config,
the data would instead be loaded based on the first quote
we received from the brokerage.
But since we're currently using a trade config, illiquid
contracts won't start streaming data until it receives a trade.
NOTE: this commit is a WIP to addressing the reviews received in the PR,
but has been committed early for efficiency in the review process
* Fixes regression algorithms and misc. bugs
* Fixes map file lookup for non-equity options
* Adds extra assertion at end of algorithm to ensure no holdings are
left when the algorithm ends.
* Adds FutureOptionSymbol, allowing all contracts through as standard
* Changes SPDB to allow defaulting to underlying future symbol
properties if no entry is found for the given FOP
* Fixes calls to SPDB in SecurityService, IBBrokerage
* Reverts AAPL daily ZIP file to fix majority of regression algorithms
* Adds FOPs symbol properties
* Fixes existing symbol properties for a few futures
* Adds tests for changes to Symbol Properties Database
* Removes string SPDB lookup method
* Updates tests and misc callees of previous method
* Updates all regression tests to use data of already expired contracts
* Adds Futures Options Expiry Functions tests
* Adds required futures data for 2020-01-05
* Address review (partial): Expands test coverage and fixes tests
* Set option chain tests parallelism to fixture only
* Fixes broken test for contract month delta for FuturesOptionsExpiryFunctions
* Changes delisting date logic for Futures Options
* Address review: removes duplicate code, misc code fixes
* Bug fix in MarketHoursDatabase.GetDatabaseSymbolKey() where
we would use the underlying's Symbol for lookup in the MHDB
* Adds missing license banner
* Removes Futures Options entries from MHDB
* Adds new tests
* Adds SecurityType.FutureOption
* Converts any underlying comparisons and uses SecurityType directly
instead for FOP specific behavior
* Extra code modifications to acommodate new SecurityType
* Addresses review: fixes order fee bug on exercise
* Additional bug fixes and adding of SecurityType.FutureOption
* Updates regression algorithms OrderListHash
* Fixes various bugs in IB live implementation
* Fixes bug setting the right contract expiration date for FOP
generated by LiveOptionChainProvider
* Adds new function to FuturesOptionsExpiryFunctions
* Clarifies parameter names better in some functions/methods
* Fixes bugs in IB brokerage for FOPs
* Address review - code cleanup and refactor
* Remove MappingEventProvider, SplitEventProvider, and
DividendEventProvider for Futures Options in
CorporateEventEnumeratorFactory
* Address review: Use MHDB key resolver in SPDB
* Makes regression tests pass and adds comment for expiry issue
* Fixes MHDB lookup on string symbol method
* Adds Futures Options greeks regression algorithm (C# only)
* Adds explanitory comment on MHDB FOP lookup
* Remove python from FutureOptionCallITMGreeksExpiryRegressionAlgorithm
* Bump time by one tick in adding subscription
* Adjust regressions
* Change removal to immediate
* Use series.AddPoint instead of directly adding it
* Adjust regression
* Add checks for fixed behavior in regressions
* Address review
* initial commit
* run parametrized algorithm with command line parameters
* skeleton: top level structure
* OptimizationNodePacket scheme
* pass parameters as HashSet
* run Lean and read results
* call method on optimization completion
* refactor public interfaces
- close ParameterSet collection; allow only get operations
- explicit method to start LeanOptimizer
* synchronize RunLean method; the result could come in before the backtest id is set in the collections
* another portion of refactoring and interface changes
* comments
* comments & tests for Extremum, Minimization and Maximization classes
* unify optimization paramater values (min, max, step) & mode GridSearch tests
- swap min&max if necessary
- iterate left => right (negate step value if necessary) & provide default step value if step == 0
- no StackOverflow Exception
- parameterSet Id should be global for current generator and retain between steps
- test signle point boundary (min == max)
* BruteForceStrategy tests
* more comments
* Update Optimizer assembly information
- Update Optimizer projects assembly information to match behavior of
the other projects
* Tweaks
- Adding comments
- Replace OnComplete for Ended event
- Replace Abort for Dispose
- ConsoleLeanOptimizer will keep track of running processes
- Each backtest will store results in a separated directory, so they
don't fight for the log.txt file.
- Adding cmdline option for lean to close automatically
- Adding concurrent execution backtest limit
- Console optimizer will start Lean minimized
- Escape spaces in Json path
* remove parameter set generator abstraction layer
we don't need this flexibility now.
* refactor public methods; Step shouldn't be public
* constraints: wip
* define contract
* comparison operators and tests
* specify JsonProperty values
* Move SafeMultiply100 to extensions
* Throw exception on failed Optimizer.Start
* constraints: wip
* change finish & dispose process
* minor fixes
- handle force lean abort
- notify consumer if target has been reached
* target & constraints; adapt unit tests
* Minor Tweaks and fixes
- Some logging improvements
- Remove Public since not required
* Ignore empty ParameterValue
* simplify condition
* avoid reinitialization
* reduce type; force immutable
* unit tests for constraints and target value
* parse & normalize percent values, i.e. 20% => 0.2
* fixup
* Target & Constraint & OptimizationNodePacket unit tests
* Add more json unit tests
- Adding more json conversion unit tests. Fix bug for Extremum which
wasn't using the converter.
* LeanOptimizer tests
* Estimation results
* User thread safe counters
* LeanOptimizer unit tests; push OptimizationResult on Ended event
* more unit tests
* Minor tweaks
-Estimate ToString in a single line.
-Typos and missing header file
* Add base SendUpdate method
- Add base SendUpdate method for LeanOptimizer
* fix LeanOptimizer test; rely on internal Update rather than timer
* Add OptimizationStatus
- Add missing commments and OptimizationStatus
* EulerSearch implementation: wip
* OptimizationParameter custom converter
* change the type
* make step optional
* change folder structure
* enumerate optimization parameter using IEnumerable & IEnumerator
* unit tests: parameters & objectives
* unit tests: strategies
* remove redundant TODO
* change Euler search boundaries
* more Euler tests
* prevent race condition
* Add account/read endpoint
- Adding account/read endpoint. Adding unit test
* Add status check before running lean
* Minor self review
- Adding missing comments, minor changes
* remove array parameters
* minor changes
- tidy up config file, rename variable
- accept min less or equal than max
* move OptimizationParameter methods to strategies
* Minor improvements for BaseResultHandler derivates
* minor changes
- strict requirements for Step and MinStep values
- strategy specific settigs
* Add TotalRuntime to estimate
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Reformat/cleanup OptionStrategies
This file was breaking pretty much every style convention in LEAN.
There are other things that should be addressed in here that weren't,
such as passing non-argument names as argument names for ArgumentException,
as well as preferring constructors over property initializer syntax, but
such changes aren't being made to keep this commit strictly reformatting
instead of refactoring.
Added braces and reformatted long lines to make code more legible.
* Add abstract base class for OptionStrategy Option/UnderlyingLegData
This allows us to create either or and later use the Invoke method to push it
into the appropriate list on OptionStrategy.
* Replace O(n) option contract search with 2 O(1) TryGetValue calls
A better improvement would be resolving the correct symbol in the strategy, but
this immediate change is instead just focused on removing the O(n) search inside
a loop.
* Add BinaryComparison and supporting methods in ExpressionBuilder
We're going to use these binary comparisons to make it possible to create
ad-hoc queries against a collection of symbols. Using these expressions,
along with type supporting composition of these expression, we'll be able
to define predicates that can declaratively define how to match an option
strategy with an algorithms current holdings.
* Make GetValueOrDefault defaultValue optional
Was receiving ambiguous invocations leading to neading to invoke this
method explicitly (LinqExtensions.GetValueOrDefault) instead of being
able to use it as an extension method. Making the default value optional
seems to have resolved this ambiguity, leading to cleaner code in the
OptionPositionCollection (forthcoming)
* Add OptionPosition and OptionPositionCollection
OptionPositionCollection aims to provide a single coherent interface
for querying an algorithm's option contract positions and the underlying
equity's position in a performant, immutable way. The immutability of
the type is necessary for how the options matcher will operate. We need
to recursively evaluate potential matches, each step down the stack removing
positions from the collection consumed by each leg matched. This will enable
parallelism of the solution as well as simplifying the mental model for
understanding due to not needing to track mutations to the collection
instance.
* Add Option test class for easily creating option symbol objects
* Add OptionStrategyLegPredicate and OptionStrategyLegDefinition
The definition is a composition of predicates, and each predicate supports
matching against a set of pre-existing legs and a current position being
checked for the next leg (this leg). In addition to the matching functionality,
it also supports filtering the OptionPositionCollection, which is where much
of the work for resolving potential option strategies is done. By successively
filtering the OptionPositionCollection through successive application of predicates,
we wil end up with a small set of remaining positions that can be individually
evaluated for best margin impacts.
All of this effectively unrolls into a giant evaluation tree. Because of this
inherent structure, common in combinatorial optimization, the OptionPositionCollection
is an immutable type to support concurrent evaluations of different branches of
the tree. For large position collections this will dramatically improve strategy
resolution times. Finally, the interface between the predicate and the positions
collection is purposefully thin and provides a target for future optimizations.
* Add OptionStrategyDefinition and OptionStrategyDefinitions pre-defined definitions
The OptionStrategyDefinition is a definitional object provided a template and functions
used to match algorithm holdings (via OptionPositionCollection) to this definition. The
definition defines a particular way in which option positions can be combined in order to
achieve a more favorable margin requirement, thereby allowing the algorithm to hold more
positions than otherwise possible. This ties into the existing OptionStrategy classes and
the end result of the matching process will be OptionStrategy instances definiing all
strategies matched according to the provided definitions.
* Add OptionStrategyMatcher and Options class, w/ supporting types
OptionStrategyMatcherOptions aims to provide some knobs and dials to control how
the matcher behaves, and more importantly, which positions get prioritized when
matching. Prioritization is controlled via two different enumerators, one controller
which definitions are matched first and the other controller which positions are
matched first. Still unimplemented, is computing multiple solutions and running the
provided objective function to determine the best match. When this gets implemented,
we'll also want to implement the timer. For anyone looking to implement these features,
please talk with Michael Handschuh as there's a particular way of representing these
types of combinatorial solutions (a 3D tree) that can be used as a variation of the
linear simplex method for optimizing combinatorial problems.
* OptionStrategyMatcher: Address PR review comments
* Ensure created OptionStrategy legs all have the same multiplier
Each leg definition match gets it's own multiplier which indicates the
maximum number of times we matched that particular leg. When we finish
matching all legs, we pick the smallest multiplier from all the legs in
the definition and use that as the definition's multiplier. When we go
to create the OptionStrategy object we MUST make sure we're using the
multiplier from the definition and not from the individual legs.
This change fixes this issue and also provides a guard clause to ensure
that we're not trying to use a multiplier larger than what was matched.
* Add XML docs for OptionStrategyDefinitions from OptionStrategies
* Fixes inability to parse negative strike prices in SecurityIdentifier
* Adds new tests ensuring backwards compat and no throwing w/ negative
strike prices
* Changes strategy used to support negative strike prices
* We add support for negative strike prices at the cost of
reducing the maximum allowed precision for the strike price.
We encode a negative sign into the 20th bit of the strike price
and set our bounds for precision to a max (exclusive) of 475712.
This in turn is then used to form a negative strike when rebuilding
the SID.
* Adds tests covering changes
* Address review: adds additional tests and refactors code
* Address self-review: remove unused import in SymbolTests
* Address review: adds additional test cases for OptionStyle and OptionRight
* These tests are to ensure that backwards compatibility is maintained
* Addresses review: Adds option chain <-> master SID hash test
* Refactors previous tests to reduce on code duplication
* Reduce test duplication
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Fix typo in data dir volume mount permission.
* Change default notebook launch behaviour for Mac systems.
Co-authored-by: Athon Millane <athon@maxkelsen.com>
* Reenable extended market hours sampling
- Will re enable extended market hours sampling using the last benchmark
and portfolio value when user exchange were open.
* Add unit test minor tweak
- Improve shutdown latency.
- Setting portfolio initial value
- Adding unit test asserting sample behavior.
* Remove python stubs directory
- Removing python stubs directory since after https://github.com/QuantConnect/Lean/pull/4899
it was been replaced by a python package `quantconnect-stubs`.
- Reverting IDE settings using the stubs folder PR https://github.com/QuantConnect/Lean/pull/4657
* Revert "Adds Python stubs location definition for PyCharm and Visual Studio Code (#4657)"
This reverts commit aded66ec5b.
* Address self-review: Provide list of imports and refactor python readme
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
* Update crypto entries in SPBD
- Update values for GDAX and Bitfinex, with latest values from exchanges
- Update symbols with >3 letters
- [Fix] Remove same base-quote entries
* Update unit tests
Increment in precision given smaller lot sizes in SPDB crypto entries.
* Add GDAX symbol properties downloader (unit test)
* Update GDAX symbol properties database
* Add BrokerageSymbol to symbol properties database
* Update GDAX symbol properties
* Address review
- Rename BrokerageSymbol to MarketTicker
* Add GDAX symbol mapper
* Update GDAXBrokerage to use symbol mapper
* Fix GDAX brokerage unit tests
* Replace GDAXSymbolMapper with SymbolPropertiesDatabaseSymbolMapper
* Address review
- use Symbol key in dictionaries
* Rename BrokerageSymbol to MarketTicker
* Save GDAX history in the real ticker folder
* rename tickerMapper to symbolMapper
* fix gdaxdownloader help message
* Save GDAX history in the real ticker folder
* rename tickerMapper to symbolMapper
* fix gdaxdownloader help message
* use SymbolPropertiesDatabaseSymbolMapper
* address review
Co-authored-by: JJD <jjdambrosio@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Update crypto entries in SPBD
- Update values for GDAX and Bitfinex, with latest values from exchanges
- Update symbols with >3 letters
- [Fix] Remove same base-quote entries
* Update unit tests
Increment in precision given smaller lot sizes in SPDB crypto entries.
* Add GDAX symbol properties downloader (unit test)
* Update GDAX symbol properties database
* Add BrokerageSymbol to symbol properties database
* Update GDAX symbol properties
* Address review
- Rename BrokerageSymbol to MarketTicker
* Add GDAX symbol mapper
* Update GDAXBrokerage to use symbol mapper
* Fix GDAX brokerage unit tests
* Replace GDAXSymbolMapper with SymbolPropertiesDatabaseSymbolMapper
* Address review
- use Symbol key in dictionaries
* Rename BrokerageSymbol to MarketTicker
* Update unit test
* Add Bitfinex symbol properties downloader (unit test)
* Add Bitfinex symbol market ticker to downloader
* Update Bitfinex symbol properties database
* Update BitfinexBrokerage to use SymbolPropertiesDatabaseSymbolMapper
* Remove Bitfinex test symbols from db
* Update Binance symbol properties database
* Update BinanceBrokerage to use SymbolPropertiesDatabaseSymbolMapper
* Add missing Binance test case
* Update symbol properties database
- gdax, bitfinex, binance
* Update CoinApi symbol mapper for new SPDB
* Exclude Bitfinex BCH pre-2018-fork in CoinApiSymbolMapper
* Remove unused properties
* Add CoinApi mappings
Co-authored-by: JJD <jjdambrosio@gmail.com>
* Fixes weeklies parsing, causing certain futures to be inaccessible in Algorithm
The FuturesExpiryFunction expects the contract month of the Future,
not the expiration. As a result, the contract gets filtered as a
weekly contract, rather than as a standard due to the discrepancy
between the expiry dates when the contract month differs from the
expiry date's month.
A very important fact to note is that futures can and do expire prior
to the contract month. BZ,(brent crude financial futures) expire two
months prior to the contract month, CL one month prior, etc.
There has been an addition that contains a "reverse" futures expiry function
lookup table. We use this to lookup the contract month to re-calculate
the Future expiry.
This PR also fixes dairy and adds extra expiry dates. Dairy can have
an expiry *after* the contract month, so a new path was added to the
SymbolRepresentation to ensure that these contracts are loaded
correctly.
* Address review: Adds tests and fixes bug in SymbolRepresentation
* Updates SID comment on `Date` property to reflect fact that we use
future expiry for its value
* Fixes bug in SymbolRepresentation where expiration day would always
be 01 when parsing a contract with an expiration after the contract
month
* Fixes bug in SybmolRepresentation where expiration year would be
four digits long when parsing a contract with an expiration after
the contract month
* Fixes some bad dairy expiry dates
* Adds tests for SymbolRepresentation and the futures filtering for
standard contracts
* Renames method used to extract delta between contract month and
expiry date
* Removes GH comment and restores Futures contract month expiry param
* Limit rounding interval to 1 day, add SubtractRoundDown and AddRoundUp functions
* fix error message
* Only subtract round down if period is greater than a day
* Tests
* Comment clarification
* Pivot solution to simple fix
* fix error message
* Remove RoundDown/Up limitations; add remarks
* address review
* Fixes 4815 by loading the requested assembly from different folder.
# Conflicts:
# ToolBox/Program.cs
* Upgrades System.Collections.Immutable to Version=1.2.5.0
* Creates a prototype for SSE streaming in IEXDataQueueHandler.
* Revert the changes in Tick.cs
* Implements a logic in IEXDataQueueHandler that updates the data-feed subscription after Subscribe/Unsubscribe
* Implements IEXCouldSubscribeMoreThan100Symbols - which fails and other small fixes.
* Implements DoForEach LinqExtensions
* Implements IEXEventSourceCollection that wraps all logic that is SSE-subscriptions and symbol-limits-per-connection concerned.
* Changes:
1) Fixes to address review.
2) Makes IexMarketPercent in QuoteSSE nullable as null values are assigned to in this field in data object received before the traing session start.
3) Deprecates helper Subscribe/Unsubscribe in IEXDataQueueHandler and IEXCouldSubscribe test.
* Fixes:
1) _refreshEvent.Reset() order was not correct - should be called before UpdateSubscription
2) ProcessJsonObject- leaves only the functionality to emit ticks.
3) IEXEventSourceCollection - replaces int counter with CountdownEvent to improve the logic - in particular, need a mechanism that would not allow the repeated call to continue until the first one is completed
* Refines the logic with parsing a data snapshot.
* Fixes few more bugs:
1) Logic in ProcessJsonObject
2) Logic in UpdateSubscription - need to introduce additional ManualResetEvent to implement the intended logic - otherwise the logic is not suitable for general case
* Introduce rate-gate limit in IEXEventSourceCollection:
because when subscribing to a bunch of shares (more than 200 for example)
the violation of rate gate policy may occur, which described in API docs asRequest Limits
IEX Cloud only applies request limits per IP address to ensure system stability.
We limit requests to 100 per second per IP measured in milliseconds, so no more than 1 request per 10 milliseconds.
SSE endpoints are limited to 50 symbols per connection. You can make multiple connections if you need to consume more than 50 symbols.:
* Few additional fixes done after real time testing
* Adds xml-docs in stream response object + renaming a file.
* Fixes:
1) Additional StreamResponseStocksUS parsing issues, that can happen outside of regular exchange hours.
2) Cancel clientUpdateThread by means of CancellationTokenSource
3) Replace BuildSymbolsQuery by string.Join
* Fixes:
1) Changing Log Trace -> Debug
2) Adds ConfigureAwait(false) to async method call
3) Removes direct reference to System.Net.Http
* Removes a task and manual reset event in IEXEventSourceCollection
* Additions:
1) IEXEventSourceCollectionSubscribes test
2) GetSnpStocksArray() helper method
3) Installs packages in QC.tests : HtmlAgilityPack & LaunchDarkly.EventSource
* IEX history provider fixes :
1) Tiny bug in ProcessJsonObject - use continue instead of return as execution is inside the for-each block)
2) Adds period variable for the historical data retrieved
3) Fixing from if (date.Date < start.Date || date.Date > end.Date) conditional check --> if (date < start || date > end) for more precise sorting.
* Changes:
1) Removes HtmlAgilityPack and SNP scraper
2) Uses hard coded symbols instead
* Bug fix:
- at certain hours (example: before pre-market open or on holidays) IEX may send no data on subscription - when trying to connect during those hours Message handler may not be fired - need to place the counter signal to client.Opened to be informed of successful connect.
* Implements:
1) IEXEventSourceCollectionSubscriptionThoroughTest and MockedIEXEventSourceCollection
2) Makes changes to IEXEventSourceCollection accordingly to allow the thorough testing.
* Fixes formatting issue in StreamResponseStocksUS
* Small fix for a new tests:
- Change RemovedClientSymbols to keep not clients itself, but symbols array, because clients are being disposed right further
* Enables extended logging in Toolbox.
* Fixing IEX historical data fetcher bugs:
1) Bug in IEXDataDownloader.cs - HistoryRequest not precisely correct.
2) Enables day-by-day daily bar downloading in IEXDataQueueHandler.
Motivation: Suppose we need data for some interval in the past - from-date=20170915-00:00:00 --to-date=20171103-00:00:00.
With current behavior IEX would have to download all the historical data from =20170915-00:00:00 up to this day.
3) Extends SynchronizingHistoryProvider
* Enables async fashion historical data download
* More fixes to IEXCouldGetHistory test.
* Reverts day-by-day daily bar downloading and other fixes.
* Removes needless packages & references
* Fix package reference
* To address review
* Sort out zero price ticks:
after testing on real-time algo 30 min before the market open now - IEX may send updates for many securities with zero lastPrice, lastSize - fix to sort such entries out
* Workaround for missing QuoteTicks timestamps:
Since we don't have a stamp for quote tick updates (only for trades) we calculate the average delay between trade tick's time stamp and local time, and
assuming that delay in average is the same for quote updates - just assign the local machine time adjusted for this average
* Simplifies the things.
* Changes:
1) Deprecates quote updates for IEX stocks.
2) Reduce the stream updates to reduce costs to ->
# Stock Quotes every 1 second (per symbol? )
# Can be up to 54,000 messages per symbol per day
https://iexcloud.io/docs/api/#how-messages-work
* Fixes:
1) IEXDataQueueHandler: give an error message on extended market hours or tick resolution subscription request. As they are not really well supported by IEX.
2) Few small fixes in IEXEventSourceCollection, including additional condition for when the subscription remains irrevocable.
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Delete QuiverHouseDataDownloader.cs
* Delete QuiverSenateDataDownloader.cs
* Delete QuiverPoliticalBetaDataDownloader.cs
* Add files via upload
* Delete QuiverHouse.cs
* Delete QuiverSenate.cs
* Delete QuiverPoliticalBeta.cs
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Delete QuiverDataAlgorithm.cs
* Add files via upload
* Add files via upload
* Add files via upload
* Addresses self review: Cleans up code and adds new unit tests
* Adds Quiver* C# files to project
* Adds new unit test for QuiverCongress
* Adds Python algorithm example
* Address self reviews
- Adding some missing xml docs
- Removing unrequired imports.
- Minor rename from Date to ReportDate
- Live trading will throw InvalidOperationException
* Fixes for example algorithms
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Add offset capabilities and related tests
* Add tests for Weekend Offsets on Symbol DateRules
* Unify Iterator Behavior
* Refactor and consolidate functions to reduce duplicate code
* Positive offset values only
* Address review
* Expand Month tests to include Forex and Crypto cases
* Ensure order of days in schedule
* Refactor and unify behavior
* More edge cases and tuning
* Address review + more tests
* Refactor and hotfix run script
Removed dependency on `realpath`, notably absent from macOS and some debian distros. Replaced with an equivalent bash function.
Cleaned up prompt response handling, replaced var tests with param expressions.
Removed potentially uneeded calls to sudo (again absent on some systems and most users are part of the docker group anyway). Instead, test if it's needed and get preauth with sudo -v.
Poll for running and stopped Lean containers, and prompt before replacing them. Would fail prior.
Uppercased variabled. Sorry.
* Revert docker output redirection to stderr
* Adjustments to maintain all functionality
* Mimic behavior across run scripts
* Update readme to reflect Docker script changes
* Ignore results storage
* Correct print statement
* Mirror changes on research docker scripts
* Make executable
* Handle already running container and sudo privs
* Fix for issues found in testing
* Address Review
* Small adjustments found in linux testing
* Doc improvement
* Add auto update option for Docker images
* Fix image var reference
Co-authored-by: Peter Kazazes <peter@peterk.co>
* DividedEventProvider distribution computation
- Update regression algorithm which was using a different reference
price when calculating the dividend
- Adjust divided event provider to compute distribution using factor
file reference price, if not 0. Adding unit tests
- For equities, only emit auxiliary data points for
TradeBar configurations, not for QuoteBars, nor internal.
* Address reviews
- Split and Dividend event provider will throw an exception when there
is no reference price available. Updating `wm` factor file which was
missing references price and regression algorithms using WM.
- Updating unit tests asserting new exception
* Protobuf will use recyclable memory stream
- Serialization will reuse recyclable memory streams
- Remove serialization of exchange and sale condition for ticks.
Updating unit tests.
- There is no need to serialize BaseData.EndTime, covered by unit tests.
* Tick will keep a parsed sale condition property
* Readd tick exchange. Json ignore ParsedSaleCondition
- name:Run Virtual Environment Test System Packages
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonVirtualEnvironmentTests.AssertVirtualEnvironment"
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonVirtualEnvironmentTests.AssertVirtualEnvironment"
- name:Run Python Package Tests
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests"
- name:Run Pomegranate Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.PomegranateTest"
- name:Run Tensorforce Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.TensorforceTests"
- name:Run StableBaselines Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.StableBaselinesTest"
- name:Run AxPlatform Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.AxPlatformTest"
- name:Run NeuralTangents Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.NeuralTangentsTest"
- name:Run NBeats Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.NBeatsTest"
- name:Run Tensorly Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.TensorlyTest"
- name:Run Ignite Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.IgniteTest"
- name:Run Hvplot Python Package Test
run:dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.HvplotTest"
<h1>Develop Algorithms Locally, Run in Container</h1>
We have set up a relatively easy way to develop algorithms in your local IDE and push them into the container to be run and debugged.
Before we can use this method with Windows or Mac OS we need to share the Lean directory with Docker.
<br />
<h2>Activate File Sharing for Docker:</h2>
* Windows:
* [Guide to sharing](https://docs.docker.com/docker-for-windows/#file-sharing)
* Share the LEAN root directory with docker
* Mac:
* [Guide to sharing](https://docs.docker.com/docker-for-mac/#file-sharing)
* Share the LEAN root directory with docker
* Linux:
* (No setup required)
<br />
<h2>Lean Configuration</h2>
Next we need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set. Just like running lean locally the config must reflect what we want Lean to run.
You configuration file should look something like this:
Our specific configuration binds the Algorithm.Python directory to the container by default so any algorithm you would like to run should be in that directory. Please ensure your algorithm location looks just the same as the example above. If you want to use a different location refer to the section bellow on setting that argument for the container and make sure your config.json also reflects this.
<br />
<h2>Running Lean in the Container</h2>
This section will cover how to actually launch Lean in the container with your desired configuration.
From a terminal; Pycharm has a built in terminal on the bottom taskbar labeled **Terminal**; launch the run_docker.bat/.sh script; there are a few choices on how to launch this:
1. Launch with no parameters and answer the questions regarding configuration (Press enter for defaults)
* Would you like to debug C#? (Requires mono debugger attachment) [default: N]:
2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
image=quantconnect/lean:latest
config_file=
data_dir=
results_dir=
debugging=
python_dir=
3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat debugging=y_**
* Accepted args for inline include all listed in the file in #2; must follow the **key=value** format
<br />
<h1>Debugging Python</h1>
Debugging your Python algorithms requires an extra step within your configuration and inside of PyCharm. Thankfully we were able to configure the PyCharm launch configurations to take care of most of the work for you!
<br />
<h2>Modifying the Configuration</h2>
First in order to debug a Python algorithm in Pycharm we must make the following change to our configuration (Launcher\config.json) under the comment debugging configuration:
"debugging": true,
"debugging-method": "PyCharm",
In setting this we are telling Lean to reach out and create a debugger connection using PyCharm’s PyDevd debugger server. Once this is set Lean will **always** attempt to connect to a debugger server on launch. **If you are no longer debugging set “debugging” to false.**
<br />
<h2>Using PyCharm Launch Options</h2>
Now that Lean is configured for the debugger we can make use of the programmed launch options to connect.
**<h3>Container (Recommended)</h3>**
To debug inside of the container we must first start the debugger server in Pycharm, to do this use the drop down configuration “Debug in Container” and launch the debugger. Be sure to set some breakpoints in your algorithms!
Then we will need to launch the container, follow the steps described in the section “[Running Lean in the Container](#Running-Lean-in-the-Container)”. After launching the container the debugging configuration will take effect and it will connect to the debug server where you can begin debugging your algorithm.
**<h3>Local</h3>**
To debug locally we must run the program locally. First, just as the container setup, start the PyCharm debugger server by running the “Debug Local” configuration.
Then start the program locally by whatever means you typically use, such as Mono, directly running the program at **QuantConnect.Lean.Launcher.exe**, etc. Once the program is running it will make the connection to your PyCharm debugger server where you can begin debugging your algorithm.
<h1>Local Development & Docker Integration with Visual Studio</h1>
<h1>Local Development with Visual Studio</h1>
This document contains information regarding ways to use Visual Studio to work with the Lean engine, this includes a couple options that make lean easy to develop on any machine:
This document contains information regarding ways to use Visual Studio to work with the Lean's Docker image.
- Using Lean CLI -> A great tool for working with your algorithms locally, while still being able to deploy to the cloud and have access to Lean data. It is also able to run algorithms locally through our official docker images **Recommended for algorithm development.
- Locally installing all dependencies to run Lean with Visual Studio on your OS.
<br />
<h1>Getting Setup</h1>
<h1>Setup</h1>
<h2>Option 1: Lean CLI</h2>
Before anything we need to ensure a few things have been done:
1. Get [Visual Studio](https://code.visualstudio.com/download)
* Get the Extension [VSMonoDebugger](https://marketplace.visualstudio.com/items?itemName=GordianDotNet.VSMonoDebugger0d62) for C# Debugging
2. Get [Docker](https://docs.docker.com/get-docker/):
* Follow the instructions for your Operating System
* New to Docker? Try docker getting-started
3. Pull Lean’s latest image from a terminal
*_docker pull quantconnect/lean_
4. Get Lean into Visual Studio
* Download the repo or clone it using: _git clone[ https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)_
* Open the solution **QuantConnect.Lean.sln** using Visual Studio
To use Lean CLI follow the instructions for installation and tutorial for usage in our [documentation](https://www.quantconnect.com/docs/v2/lean-cli/getting-started/lean-cli).
<br />
<h1>Develop Algorithms Locally, Run in Container</h1>
<h2>Option 2: Install Locally</h2>
1. Install [.Net 6](https://dotnet.microsoft.com/download) for the project
We have set up a relatively easy way to develop algorithms in your local IDE and push them into the container to be run and debugged.
2. (Optional) Get [Python 3.6.8](https://www.python.org/downloads/release/python-368/) for running Python algorithms
- Follow Python instructions [here](https://github.com/QuantConnect/Lean/tree/master/Algorithm.Python#installing-python-36) for your platform
Before we can use this method with Windows or Mac OS we need to share the Lean directory with Docker.
3. Get [Visual Studio](https://visualstudio.microsoft.com/vs/)
4. Get Lean into VS
- Download the repo or clone it using: _git clone [https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)_
- Open the project file with VS (QuantConnect.Lean.sln)
Your environment is prepared and ready to run lean
<br />
<h2>Activate File Sharing for Docker:</h2>
<h1>How to use Lean</h1>
* Windows:
* [Guide to sharing](https://docs.docker.com/docker-for-windows/#file-sharing)
* Share the LEAN root directory with docker
* Mac:
* [Guide to sharing](https://docs.docker.com/docker-for-mac/#file-sharing)
* Share the LEAN root directory with docker
* Linux:
* (No setup required)
This section will cover configuring, launching and debugging lean. This is only applicable to option 2 from above. This does not apply to Lean CLI, please refer to [CLI documentation](https://www.quantconnect.com/docs/v2/lean-cli/getting-started/lean-cli)
<br />
<h2>Lean Configuration</h2>
<h2>Configuration</h2>
Next we need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set. Just like running lean locally the config must reflect what we want Lean to run.
We need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set.
You configuration file should look something like this for the following languages:
Your configuration file should look something like this for the following languages:
<h3>Python:</h3>
@@ -78,59 +63,11 @@ You configuration file should look something like this for the following languag
<br />
<h2>Important Note About C#</h2>
<h2>Launching Lean</h2>
In order to use a custom C# algorithm, the C# file must be compiled before running in the docker, as it is compiled into the file **"QuantConnect.Algorithm.CSharp.dll"**. Any new C# files will need to be added to the csproj compile list before it will compile, check **Algorithm.CSharp/QuantConnect.Algorithm.CSharp.csproj** for all algorithms that are compiled. Once there is an entry for your algorithm the project can be compiled by using **Build > Build Solution**.
If you would like to debug this file in the docker container one small change to the solutions target build is required.
1. Right click on the solution **QuantConnect.Lean** in the _Solution Explorer_
2. Select **Properties**
3. For project entry **QuantConnect.Algorithm.CSharp** change the configuration to **DebugDocker**
4. Select **Apply** and close out of the window.
5. Build the project at least once before running the docker.
Now that lean is configured we can launch. Use Visual Studio's run option, Make sure QuantConnect.Lean.Launcher is selected as the launch project. Any breakpoints in Lean C# will be triggered.
<br />
<h2>Running Lean in the Container</h2>
This section will cover how to actually launch Lean in the container with your desired configuration.
From a terminal launch the run_docker.bat/.sh script; there are a few choices on how to launch this:
1. Launch with no parameters and answer the questions regarding configuration (Press enter for defaults)
* Would you like to debug C#? (Requires mono debugger attachment) [default: N]:
2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
image=quantconnect/lean:latest
config_file=
data_dir=
results_dir=
debugging=
python_dir=
3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat debugging=y_**
* Accepted args for inline include all listed in the file in #2
<br />
<h1>Connecting to Mono Debugger</h1>
If you launch the script with debugging set to **yes** (y), then you will need to connect to the debugging server with the mono extension that you installed in the setup stage.
To setup the extension do the following:
* Go to **Extensions > Mono > Settings...**
* Enter the following for the settings:
* Remote Host IP: 127.0.0.1
* Remote Host Port: 55555
* Mono Debug Port: 55555
* Click **Save** and then close the extension settings
Now that the extension is setup use it to connect to the Docker container by using:
***Extensions > Mono > Attach to mono debugger**
The program should then launch and trigger any breakpoints you have set in your C# Algorithm.
<h1>Common Issues</h1>
Here we will cover some common issues with setting this up. Feel free to contribute to this section!
<h1>Local Development & Docker Integration with Visual Studio Code</h1>
This document contains information regarding ways to use Visual Studio Code to work with the Lean engine, this includes a couple options that make lean easy to develop on any machine:
This document contains information regarding ways to use Visual Studio Code to work with the Lean engine, this includes using Lean’s Docker image in conjunction with local development as well as running Lean locally.
- Using Lean CLI -> A great tool for working with your algorithms locally, while still being able to deploy to the cloud and have access to Lean data. It is also able to run algorithms locally through our official docker images **Recommended for algorithm development.
- Using a Lean Dev container -> A docker environment with all dependencies pre-installed to allow seamless Lean development across platforms. Great for open source contributors.
- Locally installing all dependencies to run Lean with Visual Studio Code on your OS.
<br />
<h1>Getting Setup</h1>
<h1>Setup</h1>
<h2>Option 1: Lean CLI</h2>
Before anything we need to ensure a few things have been done:
To use Lean CLI follow the instructions for installation and tutorial for usage in our [documentation](https://www.quantconnect.com/docs/v2/lean-cli/key-concepts/getting-started)
<br />
<h2>Option 2: Lean Development Container</h2>
Before anything we need to ensure a few things have been done for either option:
1. Get [Visual Studio Code](https://code.visualstudio.com/download)
* Get the Extension [Mono Debug](https://marketplace.visualstudio.com/items?itemName=ms-vscode.mono-debug) for C# Debugging
* Get the Extension [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for Python Debugging
- Get [Remote Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) Extension
2. Get [Docker](https://docs.docker.com/get-docker/):
* Follow the instructions for your Operating System
* New to Docker? Try docker getting-started
- Follow the instructions for your Operating System
- New to Docker? Try [docker getting-started](https://docs.docker.com/get-started/)
3.Install a compiler for the project **(Only needed for C# Debugging or Running Locally)**
* Visual Studio comes packed with msbuild or download without VS [here](https://visualstudio.microsoft.com/downloads/?q=build+tools)
* Put msbuild on your system path and test with command: _msbuild -version_
3.Pull Lean’s latest research image from a terminal
-`docker pull quantconnect/research:latest`
4.Pull Lean’s latest image from a terminal
*_docker pull quantconnect/lean_
4.Get Lean into VS Code
-Download the repo or clone it using: `git clone [https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)`
- Open the folder using VS Code
5.Get Lean into VS Code
* Download the repo or clone it using: _git clone[ https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)_
* Open the folder using VS Code
5.Open Development Container
-In VS Code, either:
-Select "Reopen in Container" from pop up box.
OR
- Ctrl+Shift+P (Command Palette) and select "Remote-Containers: Rebuild and Reopen in Container"
You should now be in the development container, give VS Code a moment to prepare and you will be ready to go!
If you would like to mount any additional local files to your container, checkout [devcontainer.json "mounts" section](https://containers.dev/implementors/json_reference/) for an example! Upon any mount changes you must rebuild the container using Command Palette as in step 5.
<br />
<h1>Develop Algorithms Locally, Run in Container</h1>
<h2>Option 3: Install Dependencies Locally</h2>
1. Install [.NET 6](https://dotnet.microsoft.com/en-us/download/dotnet/6.0) for the project
We have set up a relatively easy way to develop algorithms in your local IDE and push them into the container to be run and debugged.
2. (Optional) Get [Python 3.8.13](https://www.python.org/downloads/release/python-3813/) for running Python algorithms
- Follow Python instructions [here](https://github.com/QuantConnect/Lean/tree/master/Algorithm.Python#installing-python-38) for your platform
Before we can use this method with Windows or Mac OS we need to share the Lean directory with Docker.
3. Get [Visual Studio Code](https://code.visualstudio.com/download)
- Get the Extension [C#](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) for C# Debugging
- Get the Extension [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for Python Debugging
4. Get Lean into VS Code
- Download the repo or clone it using: `git clone [https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)`
- Open the folder using VS Code
Your environment is prepared and ready to run Lean.
<br />
<h2>Activate File Sharing for Docker:</h2>
<h1>How to use Lean</h1>
* Windows:
* [Guide to sharing](https://docs.docker.com/docker-for-windows/#file-sharing)
* Share the LEAN root directory with docker
* Mac:
* [Guide to sharing](https://docs.docker.com/docker-for-mac/#file-sharing)
* Share the LEAN root directory with docker
* Linux:
* (No setup required)
This section will cover configuring, building, launching and debugging lean. This is only applicable to option 2 from above. This does not apply to Lean CLI, please refer to [CLI documentation](https://www.quantconnect.com/docs/v2/lean-cli/key-concepts/getting-started)
<br />
<h2>Lean Configuration</h2>
<h2>Configuration</h2>
Next we need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set. Just like running lean locally the config must reflect what we want Lean to run.
We need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set.
You configuration file should look something like this for the following languages:
Your configuration file should look something like this for the following languages:
<h3>Python:</h3>
@@ -84,99 +96,48 @@ You configuration file should look something like this for the following languag
In order to use a custom C# algorithm, the C# file must be compiled before running in the docker, as it is compiled into the file "QuantConnect.Algorithm.CSharp.dll". Any new C# files will need to be added to the csproj compile list before it will compile, check Algorithm.CSharp/QuantConnect.Algorithm.CSharp.csproj for all algorithms that are compiled. Once there is an entry for your algorithm the project can be compiled by using the “build” task under _“Terminal” > “Run Build Task”._
Before running Lean, we must build the project. Currently the VS Code task will automatically build before launching. But find more information below about how to trigger building manually.
Python **does not** have this requirement as the engine will compile it on the fly.
In VS Code run build task (Ctrl+Shift+B or "Terminal" dropdown); there are a few options:
- __Build__ - basic build task, just builds Lean once
- __Rebuild__ - rebuild task, completely rebuilds the project. Use if having issues with debugging symbols being loaded for your algorithms.
- __Clean__ - deletes out all project build files
<br />
<h2>Running Lean in the Container</h2>
<h2>Launching Lean</h2>
This section will cover how to actually launch Lean in the container with your desired configuration.
Now that lean is configured and built we can launch Lean. Under "Run & Debug" use the launch option "Launch". This will start Lean with C# debugging. Any breakpoints in Lean C# will be triggered.
<br />
<h3>Option 1 (Recommended)</h3>
<h2>Debugging Python</h2>
In VS Code click on the debug/run icon on the left toolbar, at the top you should see a drop down menu with launch options, be sure to select **Debug in Container**. This option will kick off a launch script that will start the docker. With this specific launch option the parameters are already configured in VS Codes **tasks.json** under the **run-docker** task args. These set arguements are:
* Would you like to debug C#? (Requires mono debugger attachment) [default: N]:
2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
image=quantconnect/lean:latest
config_file=
data_dir=
results_dir=
debugging=
python_dir=
3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat debugging=y_**
* Accepted args for inline include all listed in the file in #2
<br />
<h1>Debugging Python</h1>
Python algorithms require a little extra work in order to be able to debug them locally or in the container. Thankfully we were able to configure VS code tasks to take care of the work for you! Follow the steps below to get Python debugging working.
<br />
<h2>Modifying the Configuration</h2>
<h3>Modifying the Configuration</h3>
First in order to debug a Python algorithm in VS Code we must make the following change to our configuration (Launcher\config.json) under the comment debugging configuration:
"debugging": true,
"debugging-method": "PTVSD",
"debugging-method": "DebugPy",
In setting this we are telling Lean to expect a debugger connection using ‘Python Tools for Visual Studio Debugger’. Once this is set Lean will stop upon initialization and await a connection to the debugger via port 5678.
<br />
<h2>Using VS Code Launch Options to Connect</h2>
<h3>Using VS Code Launch Options to Connect</h3>
Now that Lean is configured for the python debugger we can make use of the programmed launch options to connect.
Now that Lean is configured for the python debugger we can make use of the programmed launch options to connect to Lean during runtime.
<br />
<h3>Container</h3>
To debug inside of the container we must first start the container, follow the steps described in the section “[Running Lean in the Container](#Running-Lean-in-the-Container)”. Once the container is started you should see the messages in Figure 2.
If the message is displayed, use the same drop down for “Debug in Container” and select “Attach to Python (Container)”. Then press run, VS Code will now enter and debug any breakpoints you have set in your Python algorithm.
<br />
<h3>Local</h3>
To debug locally we must run the program locally using the programmed task found under Terminal > Run Task > “Run Application”. Once Lean is started you should see the messages in Figure 2.
If the message is displayed, use the launch option “Attach to Python (Local)”. Then press run, VS Code will now enter and debug any breakpoints you have set in your python algorithm.
Start Lean using the "Launch" option covered above. Once Lean starts you should see the messages in figure 2 If the message is displayed, use the launch option “Attach to Python”. Then press run, VS Code will now enter and debug any breakpoints you have set in your python algorithm.
Here we will cover some common issues with setting this up. This section will expand as we get user feedback!
* Error messages about build in VSCode points to comments in JSON. Either select **ignore** or follow steps described [here](https://stackoverflow.com/questions/47834825/in-vs-code-disable-error-comments-are-not-permitted-in-json) to remove the errors entirely.
- The "project file cannot be loaded" and "nuget packages not found" errors occurs when the project files are open by another process in the host. Closing all applications and/or restarting the computer solve the issue.
- Autocomplete and reference finding with omnisharp can sometimes bug, if this occurs use the command palette to restart omnisharp. (Ctrl+Shift+P "OmniSharp: Restart OmniSharp")
- Any error messages about building in VSCode that point to comments in JSON. Either select **ignore** or follow steps described [here](https://stackoverflow.com/questions/47834825/in-vs-code-disable-error-comments-are-not-permitted-in-json) to remove the errors entirely.
thrownewException($"{Time}:: Detected differences in expected and actual securities{Environment.NewLine}Expected:{Environment.NewLine}{expected}{Environment.NewLine}Actual:{Environment.NewLine}{actual}");
thrownewException($"{Time}:: Detected differences in expected and actual universes{Environment.NewLine}Expected:{Environment.NewLine}{expected}{Environment.NewLine}Actual:{Environment.NewLine}{actual}");
thrownewException($"Expected Symbols selected on {Time.Date:yyyy-MM-dd} to match expected Symbols, but the following Symbols were missing: {string.Join(",", missing.Select(s => s.ToString()))}");
}
_coarseSelected[Time.Date]=true;
returnselectedSymbols;
}
publicoverridevoidOnEndOfAlgorithm()
{
if(!_coarseSelected.Values.All(x=>x))
{
thrownewAggregateException($"Expected coarse selection on all dates, but didn't run on: {string.Join(",", _coarseSelected.Where(kvp => !kvp.Value).Select(kvp => kvp.Key.ToStringInvariant("yyyy-MM-dd")))}");
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the evemts</param>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the evemts</param>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the evemts</param>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the evemts</param>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the evemts</param>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.