* Fix fill forward resolution adjustment on removal
- Fix fill forward resolution adjustment on subscription removal. Adding
regression algorithm reproducing issue
* Minor fix for undeterminism behavior on security removals and FF res change
* Further improvements and determinism
- Improve regression algorithm add, remove and readd future, asserting
FF resolution changes
- Adjust universe selection to be deterministic and avoid unnecessary
FF resolution changes
* Daily data Time & EndTime Improvement
- Adjust daily data Time & EndTime to actually reflect the time of the
data used, for example US Equity from 9.30 to 4PM. Adding new unit and
regression tests
* Refactor solution to use enumerator
- Refactor daily strict end times solution to be through enumerator
usage, so it applies for history providers too
* Minor fixes
* Revert fill forward enumerator change
- Revert FillForward enumerator causing stats changing, enhancing unit
tests
* Some cleanup
* Improve handling of live trading FF enumerator
- Improve handling of live trading FF enumerator, by adding support for
bars to arrive with a delay so we can handle auction close/option
prices or data providers which might have some delay making the data
available. Adding new unit tests asserting the behavior
* Round combo order prices
* Combo limit order round fix: use smallest security min price variation
* Minor unit test change
* Improvements and unit tests
* Minor change
* Cleanup
* Handle expired options
* unit test
* Move Zero Checks to Single Location
---------
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
* Fix: python consolidator reference lost when wrapped
* Make BasePythonWrapper equatable depending on the PyObject underlying reference
* Fix: TryConvert PyObject instances for generic C# types
* Minor unit tests fixes
* Address peer review and add unit tests
* Fix: properly adjust prices after a split
* Minor fixes
* Minor: Adjust cached data on split
* Minor change
* Expand regression algorithms assertions
* refactor: modifier static to parse enum and ignore warning
* feat: group symbol when download cannonicial data
feat: log total time execution of download provider
test: right keeping of cannonicial data
* feat: parameter ctor in DataDownloadConfig
* test:feat: additional assertion of wrote data in file
* feat: static FilterGroupDownloadDataBySymbol
* refactor: extract downloader folder path
* try: map file false
* feat: get rid of dataCacheProvider in DownloaderDataProvider
* refactor: get rid mapFile flag
* CI test fix
---------
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Fix: source custom data from remote CSV zipped file
* Fix: Custom data sourcing zip entry names from remote locations
* Address peer review
* Add unit tests
* Minor fix
* Address peer review
* Minor fixes
* Return byte array in IApi.DownloadBytes
* Allow sourcing custom data from specific csv entry in a zipped remote file
* Minor change
* feat: init Download proj
* feat: update docker file to copy download proj
* feat: parsing in cmd arguments
* feat: DataDownloadConfig model
feat: add config file
refactor: logic of parsing config
* feat: overrid ToString in DataDownloaderGetParameters
* remove: not used argument command
* feat: calculate downloading % (without ETA time)
* fix: dockfile copy path for DownloaderDataProvider proj
* feat: additional Property in .csproj
* remove: DataProviderFullName prop from DataDownloadConfig
* remove: extra package (ref from Configuration proj)
* feat: ETA calculation of downloading data
* Minor tweak
* rename: namespace of downloaderDataProvider to generic style format
* test: ETA from DownloaderDataProvider
* fix: calculate ETA like real downloading process
---------
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
- Make sure calling CSharp methods, fields, properties releases the GIL,
this was not hapenning for fiels & properties, adding unit test
reproducing the issue
* pep8 conversion of alphas
* pep8 conversion of benchmarks
* Various fixes and rebase
---------
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Initial boilerplate code
* Fix build failure
* Initial iteration for the code for Derivative Oscillator code
* Add foundation for unit tests
* Add reset method for test
* Add test case
* fixed test cases, all tests pass
* Put placeholder for IsReady change
Not implemented, as test cases fail when change is uncommented
* Update documentation and change class name
* Change names for Derivative Oscillator to more meaningful
* Add DO extension to create the indicator
* Provide variable definitions with var, replace old test file with new
* Use chaining for indicators creation
* Add test based on external data
* Change test file
* Fix WarmUp test + alphabetical order + rename indicator class
* Make all tests run except for external data tests
* Fixed warmup period and fixed the ResetsProperly test case
* Removed unused directives
* Add output directory for Derivative Oscillator test
* Minor tweaks
---------
Co-authored-by: HaentjeGH <l.m.j.haenen@student.tue.nl>
Co-authored-by: ivkozlov <ivkozlov@ozon.ru>
Co-authored-by: DanielVanRijswijk <d.j.v.Rijswijk@student.tue.nl>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Improve python exception message for cases when derived class uses member names already defined in base C# classes
* Update pythonnet version to 2.0.33
* Update error message
* Pythonnet version bump to 2.0.34. Minor unit test fix
* Enhance pythonnet overload tests
---------
Co-authored-by: Jared <jaredbroad@gmail.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
- Pep8 adjustment for GoodTilDate, so it behaves like `Expiry` static
readonly funcs which will be binded to both snake case and upper
constant format, `good_til_date` & `GOOD_TIL_DATE`
* feat: support snake-case style Python QCAlgorithm implementations
* feat: add unit tests and minor fixes
* feat: implement new BasePythonWrapper class for python wrappers.
Used to cache methods and contains invoke functionality
* feat: make python wrappers implement the new base class for pep8 style support
* feat: keep overriden methods in Algorithm Python Wrapper
* feat: add unit tests for custom models algorithms with PEP8 style
* Bump pythonnet version to 2.0.30
* fix bugs and address peer review
* Address peer review
* Minor revert
* feat: StubsIgnoreAttribute for ignoring members or classes by the stubs generator
* Minor fixes
* Minor fix
* Minor fix
* Bump pythonnet version to 2.0.31
* Added Greeks.Lambda_ alias of Lambda for python compatibility.
Remove unused method
* Add support for auxiliary data history request
- Add support for split/dividends/margin interest history requests.
Adding regression algorithms
* Expand auxiliary history regression tests
- By default Time rules yield times in UTC TZ, already correctly handled
by 'ScheduledUniverseSelectionModel' but this wasn't fixed in ScheduledUniverse
* Minor api improvements
- Adding Universe.Selected to hold the current set of selected symbols.
Adding regression tests
- Adjusting TimeRules and DateRules to fallback to the MHDB if the
security hasn't been added. Updating existing regression tests
* Expand regression test assertion
* fix: download year file data only for hour and daily resolutions for options and index options
* minor fix: index options regression tests expected status
* Minor change
* minor: update regression algorithms stats
* Adding new algorithm data handling options
- Adding new QCAlgorithm data handling method options to override.
Adding Py/C# regression tests
- Removing old OnData Type specific invoker methods
* Address reviews. Remove old OnData C# only methods
* Add Scheduled Universe Selection
- Adding `UniverseSettings.Schedule` that will allow users to
specify a custom selection schedule which is independent of the
underlying data frequency
- Adding unit and regression tests
* Add live trading schedule time shift
* Support CFDs in IB
* Add new currency pair util method
* Minor changes to IB fee model and tests
* Add CFD USA market hours
* Add new Market interactivebrokers for cfds
* Add sample algorithm illustrating manually adding market hours to MHDB
* Update MHDB and SPDB
* First draft of the solution
* Add missing changes
* Remove the new KPI's from report
* Fix bugs
* nit change
* Add improvements
* Fix regression tests
* Solve bugs in the regression algos
* Fix regression tests bugs
* Expand unit tests and add minor changes
* refactor: map files in lean/data folder
* feat: Indexers in MapFile
* feat: extension for mapFile and downloader param
* refactor: getAllTickers from MapFiles
* feat: addition out variable in TryParsePath
fix: tests with new variable
* refactor: DownloadDataProvider with getting all ticker with different DateTime Ranges
feat: test
* revert: "refactor: map files in lean/data folder"
This reverts commit 41fa26eb8627446028d4f2d4c536511ea82a9a48.
* revert: "feat: addition out variable in TryParsePath"
refactor: some actual changing
This reverts commit 9de482bd6490eecd1eeee83e035dfc1ece8d6415.
* revert: "feat: Indexers in MapFile"
fix: actual changes
This reverts commit 393181c6316a354a8cdcbfb093707cb545383a8a.
* remove: high performance Any validation
feat: validation on ticker start with specific sign
fix: option test
* feat: validation on mapping
refactor: when resolution less hour
test: new test cases + validation of SecurityType
* refactor: add additional day when return date range from MapFile
fix: FirstTicker instead of Permtick when generate new Equity
feat: validation of request Date in DownloaderDataProvider
refactor: by style proj DownloaderDataProvider
test:fix: starting Date in TestCases
* remove: duplicate code
* fix: check of income data.Time with requested Start/End-Date
* test:fix: download test cuz we can not get/write future data
* remove: extra Symbol Create mthd
* test:feat: validate of TryParse returns correct symbol
* remove: excess if condition
* test:remove: extra chaning
connected # 3a62adf54c
* refactor: split big method to small ones (add readability)
* fix: typo in xml description
* refactor: get the correct symbol value
* refactor: use local time instead of Utc in download Param extension
refactor: use struct instead of class
* test:fix: convert time to Utc from local time
* Adds Rebate and Fee Rates to Shortable Provider
If the provider does have this information, the rates are zero.
* Adds Helper Methods to QCAlgorithm
- Improve summaries.
- Express values as rates instead of percentages.
- Adds headers to files to emulate real data
* Clarify the Data Format
* Removes Helper Methods from QCAlgorithm
* Removes `Shortable` and `ShortableQuantity` from Example
Algorithms should prefer getting the information from the `Security.ShortableProvider`.
* Add schedule manager unit tests
* Add schedule manager unit tests
* Fix overriding security exchange hours
* Minor comments changes
* Minor change
* Add SecurityExchangeHours.Update method to set without changing the reference
* Minor change
* Refactor universe historical data source
- Add new universe history API methods
- Refactor QuantBook UniverseHistory to use the universe selection
itself instead of a given func
- Refactor and rename fundamental types
- Refactor AddUniverse API to handle universe collection data which
holds another type internally, like fundamental
* Fix minor bug causing ApiDataProvider not to serve Bitfinex universe data
* Further improvements to add universe API
* Handle no selection function
* feat: helper to return old name of symbols
feat: test
refactor: test a little bit neat
* feat: SplitHistoryRequest() with different mapped symbol
feat: test
refactor: RetrieveSymbolHistoricalDefinitions
* refactor: split historyRequest by historical symbol name
refactor: change location of extension
refactor: change location of test extension
* revert: extra changes
* fix: validation on RequiresMapping in SplitHistoryRequest
feat: add Future Symbol tests
* move: extension method to another file
* fix: wrong extension call param position
* fix: wrong extension call param position in test
* refactor: IEnumerable foreach to more simple form
feat: add TickerDateRange struct instead of Tuple
fix: test
* refactor: map extension alway returns empty result
* refactor: SplitHistory extension returns original HistoryRequest
* feature: new extension for RequiresMapping except Futures
* fix: validation of symbol map file to contains something
* refactor: extension test cases
* feat: assert of right ticker in tests
* fix: split historical request with using Local Time instead of UTC
fix: test description
* feat: test helper IEnumerable of TimeZones
* test: validate right EndDate with different TimeZones
* remove: validation of end of date
test: several new test cases
* feat: additional test with specific dateTime
* remove: extra Extension method validation on future and mapping
* Allow mirror contract and smoothing IV
* Python wrapper on smoothing IV
* helper method update
* Add forward tree model
* modify and add unit tests and data
* address peer review
* address peer review
* Update symbol-properties-database.csv
Updating Symbol properties database to reflect kraken API information.
#### Description
Updated the market_ticker column of Symbol properties database to reflect information as per Kraken API.
#### Related Issue
Close#7794
#### Motivation and Context
We updated symbol mapping for the Kraken market.
#### Requires Documentation Change
No
#### How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
#### Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Refactor (non-breaking change which improves implementation)
- [ ] Performance (non-breaking change which improves performance. Please add associated performance test and results)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Non-functional change (xml comments/documentation/etc)
#### Checklist:
<!--- The following is a checklist of items that MUST be completed before a PR is accepted -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [X] My code follows the code style of this project.
- [X] I have read the **CONTRIBUTING** [document](https://github.com/QuantConnect/Lean/blob/master/CONTRIBUTING.md).
- [X] I have added tests to cover my changes. <!--- If not applicable, please explain why -->
- [X] All new and existing tests passed.
- [X] My branch follows the naming convention `bug-<issue#>-<description>` or `feature-<issue#>-<description>`
<!--- Template inspired by https://www.talater.com/open-source-templates/#/page/99 -->
* Update symbol-properties-database.csv
- Update as per peer review
* Correction
* Implement VaR in PortfolioStatistics
* Added ValueAtRisk indicator
* Refactored ValueAtRiskTests
* Added returns column to test data
* Change indicator helper method name
* Addressed comments
* Chande Variable Index Dynamic Average (VIDYA) Indicator
* Improvements
* Improvements
* Final Changes
* Editted, Retested
* Comments
* Minor Corrections
* Updated Datetime Format of test data to US Standards
---------
Co-authored-by: Pronoy Chowdhury <rokstarr.dbz@outlook.com>
* Make sure continuous futures universe selection is performed right away on the start of the algorithm.
This ensures the Future.Mapped is not null
* Minor changes
* Minor fixes
* Update regression algorithms stats
* Add DC futures map file
* Update DC futures map file
* Update regression algorithms stats
* Include ETFs for immediate universe selection
* Include Coarse Fundamentals for immediate universe selection
* Include Futures Chain Universe for immediate selection
* Add Option Chain Universe test
* Include custom universes for immediate selection
* Minor updates
* Include constituents universes for immediate selection
* Minor update
* Minor changes, docs and stats updates
* Minor changes
* Minor changes and clean up
* Minor regression algorithm stat update
* Models OZN: 10-Year US Treasury Future Options
<!--- Provide a general summary of your changes in the Title above -->
#### Description
We added future option for OZN
#### Related Issue
Close#7783
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- Please link to the issue here: -->
#### Motivation and Context
We added future option supported by CME
https://www.cmegroup.com/markets/interest-rates/us-treasury/10-year-us-treasury-note.contractSpecs.options.html#optionProductId=317
#### Requires Documentation Change
No
#### How Has This Been Tested?
I tested the change using fake files to check if mapping is correct.
#### Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Refactor (non-breaking change which improves implementation)
- [ ] Performance (non-breaking change which improves performance. Please add associated performance test and results)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Non-functional change (xml comments/documentation/etc)
#### Checklist:
<!--- The following is a checklist of items that MUST be completed before a PR is accepted -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [X] My code follows the code style of this project.
- [X] I have read the **CONTRIBUTING** [document](https://github.com/QuantConnect/Lean/blob/master/CONTRIBUTING.md).
- [X] I have added tests to cover my changes. <!--- If not applicable, please explain why -->
- [X] All new and existing tests passed.
- [X] My branch follows the naming convention `bug-<issue#>-<description>` or `feature-<issue#>-<description>`
<!--- Template inspired by https://www.talater.com/open-source-templates/#/page/99 -->
* Add unit tests
---------
Co-authored-by: Ashutosh <ashutoshrana171@gmail.com>
* Improve consolidator scan
- Improve consolidator scan to be triggered even if no data is present.
Adding new regression test, updating existing
* Adding more tests & minor tweaks
- Adjust live equity delisting to be map files based like splits and
dividends. Adding unit test
- Expand existing unit tests asserting delistings in live mode, set
securities to not tradable
* Delay security removal when there are pending settlements
* Add unit test
* Minor fixes
* Minor fix
* Replace HasUnsettledFunds method with UnsettledCash in ISettlementModel
* Minor fix
* Minor fix
* Address peer review
Replace ISettlementModel UnsettledCash property with GetUnsettledCash method that returns a CashAmount instead of a decimal to include the currency
* Minor fix
* Address peer review
* Minor chart improvements
- Add base series tooltip template. Adding unit tests
- Reduce null chart points to a single addition
* Add null values always
* Fixes & minor tweaks
* Serialize ChartPoint as array
* Portfolio state chart
- Cleanup and renames
- Add series.IndexName. Uodating unit tests
- Add Chart.LegendDisabled. Adding new unit tests
- Add ChartPoint.Tooltip. Updating unit tests
- Minor compression tweak. Adding unit test
- Add ChartJsonConverter. Adding unit tests
* Minor chart serialization order tweak
* Refactor portfolio state sampling and storing
* Move PortfolioMargin into a lean side chart
* Implement backtest self-naming and self-tagging
* Minor unit test fix
* Minor api changes
* Minor changes
* Send packet messages when algorithm's name or tags are updated
* Minor fixes
* Address peer review
* Backtest name and tags API calls
Plus, first version of CI for API tests
* Add endpoint to update a backtest set of tags
* Minor GH actions change
* Address peer review
* Add algorithm naming and tagging unit tests
* Minor backtest API changes
* Models All Index Futures Listed in CME
We added:
NIY, ENY, FT1, FT5, E3G, FTU, TPD, TPY
* Models All Index Futures Listed in CME
We added international index futures:
NIY, ENY, FT5, FT1, E3G, FTU, TPD and TPY
* Remove unused imports
* Address peer review:
- Corrected serial contract logic
- Corrected minimum price fluctuation
- Corrected “Yen” to “JPY”
Added equity index futures:
RX, ESG, RS1 and SDA
* Replace currency string formatting with Lean currency symbols.
Also clean up orders tags from having redundant information.
* Setting default order tag only after price currency is properly set
* Minor fixes and new unit tests
* Minor fixes
* Fix bug in ETFConstituentUniverseSelectionModel()
The bug was raised after calling an overload constructor using just the
ticker as parameter. Since the universeFilterFunc was null, the method
Extensions.TryConvertToDelegate<T> returned true and thus, the filter
func assigned was the one returned by the method
`ConvertToUniverseSelectionSymbolDelegate()` which used the null
filterfunc.
* Handle potential future bugs
* Initial draft of the solution
* Add improvments
* nit changes
* Improve implementation
* Enhance unit tests
* Add one more unit test
* Fix failing tests and update MHDB
* Add requested changes
* Nit change
* Add tweaks to MarketHoursDatabase class
* Solve bug and add unit tests
* Update unit tests
* Address requested changes
* Handle potential bug
* Nit change
* Minor review
---------
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Handle/filter brokerage side orders
* Add unit tests
* Minor fix
* Add unrequested security for new brokerage-side orders
* Add unit test for algorithms overriding the brokerage message handler
* Add unit test for python algorithm overriding the brokerage message handler
* Minor change
* Address peer review
A more specific entry might contain some of the same early close/late open dates as the common entry but with different times. This makes sure this is handle and the more specific ones take precedence.
* MaximumSharpeRatioPortfolioOptimizerTests
* MinimumVariancePortfolioOptimizerTests
* UnconstrainedMeanVariancePortfolioOptimizerTests
* Addressed comments
* Address comments v2
* Refactored tests with base test class. Added Python wrapper tests
Base test class and python wrapper tests
* Add IV indicator
* Fix bug
* Add QCAlgorithm helper method
* Add unit tests
* Add OptionPricingModelType, as option for greeks & IV estimation
* Nit, and fix bug
* Address peer review
* Fix bug
* Fix bug on CRR
* Ensure test indicator initiate correctly
* Address 2nd review
* Check for FOPs and index options as well
* Add comparison with QuantLib
* CRR not follow BSM
* minor bug fix
* Asset plot improvement
- Add optional symbol for chart instance, to create an asset plot
- Fix Order PriceAdjustmentModel deserialization. Updating unit test
* Add series ZIndex. Adding unit test
* Fix clone, adding more tests
* feat: new `TradingDaysPerYear` prop in IAlgorithmSettings
* feat: getTradingDayPerYear by BrokerageModel
feat: init algo tradingDaysPerYear in brokerage|BacktestingSetupHandler
refactor: nullable `TradingDayPerYear`
* remove: hardcoded value:252 in statistics
* test: `algorithm.Settings.TradingDaysPerYear` for different brokerage
fix: tests calculation of `AnnualPerformance()`
* fix: Report generator by tradingDayPerYear
* fix: hardcoded value in PortfolioStatisticsTests
* fix: tradingDayPerYear in AnnualPerformanceTest
remove: extra spacing
* feat: backwards compatibility for TradingDaysPerYear
* feat: several UTest for AlgoSetting's TradingDaysPerYear prop
* fix: existed algo + test
* feat: regression algo with checking of tradingDayPerYear in OnEndOfAlgorithm()
* refactor: GetBrokerageTradingDayPerYear -> SetBrokerage... in BaseSetupHandler
refactor: carry out tradingDayPerYear at the top level and pass variable
* remove: default valuine in test param
remove: extra comment.
* fix: missed init TradingDaysPerYear in report.portfolioAlgo
fix: hardcoded value to tradingDaysPerYear
* fix: nullable TradingDaysPerYear in AlgorithmConfiguration
feat: UTest for PortfolioLooperAlgo
feat: add advanced description
* feat: test PortfolioStatistics with different tradingDaysPerYear
* revert: PortfolioLooper with TradingDaysPerYear implementation
* revert: nullable TradingDaysPerYear in AlgoConfiguration
remove: SetBrokerageTradingDayPerYear in BaseResultHandler
* style: remove empty row
* feat: stopMarketFill in FutureFillModel + test
* feat: futureStopMarket RegressionAlgo on PY|C#
* feature: add validation of market time in utest
* feature: validate exchange hours when order filled in regression algos
* feat: validation of StopMarket Type
fix: styling format in PY
* fix: validation on ExtendedMarketHours in FutureFillMode
feature: validate that all stop orders was completed in algos
* fix: validation: asset.Subscription -> ConfigProvider.GetSubscriptionData
fix:style: missed braces
* fix: asset.IsInternalFeed() -> general includeInternalConfigs flag
remove: extra login in algo
* feat: write comment why we use includeInternalConfigs in true
remove: extra log in PY version of algo
* Improve GetFillModel()
* Add support for other methods
* Add regresion algos and improvements
* Make Python algorithms regression tests
* Improve unit tests
* Minor changes
* First Draft of the solution
Handle the case where the security has already been added in the
algorithm
* Improve implementation
* Improve changes
* Add unit test
* Improve unit tests
* Fix bugs in unit tests
* Nit changes
* Nit changes
* Improve implementation
* Solve bugs in unit tests
* Add more unit tests
* Plotting Limits Handling Improvements
- Improve the handling and enforcement of plotting limits
- Increase local default limits
* Minor plotting test fix
* Solve bug and add unit tests
The bug was raised due to a race condition when writing a file and
deleting it. In order to stop it, a check condition was added at the end
of the PersistData() method to remove files that have just been deleted.
* Nit changes
* Add a different constructor for the test
The old description is incorrect. A stock can have positive Alpha but have negative Beta. In this case, if the benchmark increases, the performance of the stock may be dominated by its negative Beta, causing it to underperform the benchmark.
* Adds Unit Tests for ParseFutureSymbol
Tests:
- Ambiguous year with missing decade
- Non-ambiguous known year
- Expiring before 2000
* Fixes Missing Decade Tests
If the ticket doesn't provide a decade, VXZ2, we assume the current decade. So VXZ2 expires on Dec 2022.
* Fixes SymbolRepresentation.ParseFutureSymbol
Use the value of `futureYear` to handle contracts that expire before 2000. (E.g.: NQZ99).
Tickers from live trading may not provide futureYear. Use the current decade (2020).
- Fix bug where there might not be any data to sample. Adding unit test
- Fix bug where candlestick sampling might add point which shouldn't be
there. Adding unit tests
- Adjust series json converter to serialize any base series type. Adding
unit test
* Add Sortino ratio to statistics and report
* Adds Sortino Ratio to Report Key Statistics
* Addresses Peer-Review
Reuse `SharpeRatioReportElement` and change the template.
* Reuse Calculations Across Statistics and PortfolioStatistics
* Adds Sortino Ratio to Regression Algorithms
* Removes Sortino Ratio from Optimization Result Table
---------
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
* Add support for null chart points
- Add support for null value chart points, updating existing and adding
new unit tests
* Minor selfreview tweaks
* Address review & fix deserialization bug
* Ignore null values when using NullResultValueTypeJsonConverter
* Narrow NullResultValueTypeJsonConverter conversion support
* Allow inherit of series sampler
* Solve bug and add unit test
The bug was raised when the sum of the True Range values was zero. Since
the Ultimate Oscillator provides a more reliable indicator of momentum,
if the prices don't change, its value should be zero, as there is no
acceleration of a security's price.
* enhance implementation
* Index.IsTradable Can Be Set
Indices are not tradable. LEAN will not set it to `true`, but the users can for backtesting.
* Adds Regression Tests
* Revert Changes to ProcessSecurityChanges
After running the regression tests in my local pc, I found some of them
were failing in the stats. Then, after debugging them I realized it was
because some of the strings were being casted without using
InvariantCulture parameter.
* Update Future-cme-[*] and Future-cme-ES
Acoording to `pandas_market_calendars` there were some dates in
Future-cme-[*] who wasn't early_closes, so they needed to be removed
from there. On the other hand, the early closes list of Future-cme-ES were
shifted by 1 hour according to CME webpage. Besides, there were some
missing dates.
* Update CME Future entries in MHDB
* Rebase
* nit change
* Fix unit tests
* Resume after early close/halts
* Add missing dates in MHDB and fix bugs in it
* Fix bug, add more unit tests and add docs
* fix regression algos
* address required changes
* Update failing regression test stats
After debugging the tests it was found they were failing due to the last
change on SecurityExchangeHours.IsOpen(). That method wasn't taking into
account that even if there is a late open after an early close if the
timespan is after the early close but before the late open, the market
is still close.
* enhance solution
* Update and fix bugs in MHDB
* Address required changes and update stats
* Update stats after rebase
* Nit change
* Missing update to regression test
* Use MHDB instead of USHoliday for Expiration Dates
VIX expiry function now relies completely on MHDB. However, it had to be
created an entry in MHDB for VIX since there wasn't one for it. CBOE
webpage only provided 2023 holidays so only those dates were considered
in the Holidays entry in MHDB. Therefore, some unit tests failed so it
was necessary to change also the VIX entry in FuturesExpiryFunctionsTestData.xml.
* Remove Global.cs/USHolidays class
* Use a lazy implementation
* First draft of the solution
* Use MHDB in FuturesExpiryFunctions.cs
* Remove unused class and fix indentation errors
* Fix indentation errors
* Nit changes
* Merge branches 7501 and 7506
* Merge changes in 7501 and 7506
In order to check compatibility between those branches, a new branch
was created out of branch 7501 and then it was merged with branch 7506.
2 regression tests and 8 unit tests failed, the regression tests failed on
the DataPoint stats. On the other hand, the unit tests failed since the
default parameter UseEquityHoliday was removed from
FuturesExpirtyUtilityFunctions.AddBusinessDays() and from other methods in
the same class too.
* Add missing changes
* Remove repeated good fridays
* Address minor review
---------
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* feat: ETFConstituentsModel overloading constructor
* test
* feat: new constructor of ETFConstituentsUniverse
test: support all constructor on c#
* feat: cached symbol in ETFConstituentsUniverse.py
test: validate getting symbol from cahed collection
* remove: extra ctor in ETFConstituentsUniverseSelectionModel
* refactor: add miss condition secType in py
refactor: rename out existTicker -> Symbol
refacotr: rewrite test
* feat: new ETFConstituentsFrameworkAlgos
* refactor: more neat ETFConstituentsAlgos
* fix: CompositeUniverse to SetUniverse
* Implement risk free interest rate as an algorithm model
* Use risk free insterest rate model in Sharpe Ratio indicator
* Address peer review
Also added python wrapper
* Take pyobject as interest rate model in Sharpe Ratio indicator
* Minor fix
* Minor fix
* Address peer review
* Fix using brokerage transaction handler algo befor set
Brokerage open orders are being set before algorithm is set to the transaction handler, so SetPriceAdjustmentMode needs to get the algo to use
* Update unit tests
* Add new Order.PriceAdjustmentMode property
* Minor fix and unit test
* Minor fix and regression algorithms' stats update
* Unit test fixes
* Minor fix
* Set order price adjustment mode to raw always for live trading
* Shortable modeling improvements
- Do not limit live trading because of shortable model, will send
algorithm warning
- Add Interactive brokers shortable provider
- Minor LocalDiskShortableProvider API changes
* Fix shorted order update
- Fix shorted order update. Updating regression algorithm
* Minor improvement
* Address reviews
* Improves Error Handling in OptionPriceModelResult.Evaluate
Use the maturity calculation to calculate dividend and risk free discount to prevent the `negative time` error.
Adds Exception Message and StackTrade to `DebuggingEnabled` messages.
* Set Evaluation Date Before Every Calculation
Adds loggings to record the reference date which is set by the `Settings.setEvaluationDate` method.
- Once a security has been delisted or no longer selected by any
universe it will be remove from the main algorithm securities
collection but kept in the new Complete collection, this improves
performance in long running backtests where security collection can
get enumerated in different cases
- Minor refactor for security exchange local time to avoid having to set
it explicitly
* correlation as lean indicator #6983
* correlation as lean indicator #6983, added SPY QQQ test and Spearman test
* correlation as lean indicator #6983
* correlation as lean indicator #6983, fixes after review
* correlation as lean indicator #6983, fixes after review
* correlation as lean indicator #6983, fixes after review
* correlation as lean indicator #6983, added AcceptsQuoteBarsAsInput Test
* Minor review changes
---------
Co-authored-by: Grzegorz Wodnicki <info@funchips.co.uk>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Store traded securities subscription data configs to disk
* Delegate traded subscriptions storage to the data monitor
* Add unit tests
* Address peer review
* Minor changes
* Async universe selection
- Add support for async universe selection, which will happen ahead of
time in the data stack for a performance improvement
* Thrown if using Coarse+Fine Asynchronous Universe selectioon
* Add Bybit brokerage model regression algorithm
* Add BibytFutures brokerage model name and Bybit backtesting regression algorithms
* Add Bybit margin calculations unit tests
* Minor changes in data
* Unify Bybit Spot and Futures brokerage model into one class
* Add new Bybit configurations
* Revert config change
* New Fundamental Data
* Minor CIK lookup fix
* Handle live mode & delete unexisting properties
* Minor coarse fundamental adjustment
* Add fundamental history support
* Fix unit tests
* Performance improvements
* Fixes
* Minor regression algorithm fix
* Improvements. Add FundamentalUniverseSelectionModel
* Change default values
* Fix unit test
* Minor tweaks
* Fix unit test
* Minor error handling improvement
* Fix rebase
Adds a new constructor overload that matches the pattern in Lean: Symbol, Symbol, other types.
Adds a new constructor that infers the indicator name: "B(period)".
* Add new libraries and updates
* ARM foundation update
- Deleting creme library which has been merged into already included river, because it causes copulae build to fail
- Deleting Riskfolio-Lib which would try to install old numpy and fail
* Add ClrBubbleExceptionInterpreter
* Add ClrBubbledExceptionInterpreter tests
* Bump pythonnet version to 2.0.23
* Minor changes
* Minor change
* Minor fix
* Fix failing unit tests
* Set default always open market hours entry for base security without subscription
Also return proper matching subscription for custom data symbols wihtout subscription.
* Minor changes
* Minor change
* Adds `SimpleCustomFillModel` to `CustomModelsAlgorithm`
The simple fill model shows how to implement a simpler version of the most popular order fills: Market, Stop Market and Limit.
This model was tested on QuantConnect Cloud, and will serve as additonal example, since we don't have an example that does not reuse the method of the base class.
* Handles Tick Resolution Case
Tick-resolution data doesn't have TradeBar. We can use the security price, since it represents a trade (`TickTrade`).
* Support sourcing custom data from ObjectStore
* Add Python regression algorithm
* Wire engine to pass ObjectStore down to the stream readers
* Minor changes
* Minor unit tests fixes
* Remove unused SetupHandlerParameters.ObjectStore
* Minor changes
* Minor changes
* Support single-file zipped data to be sourced from object store
* Assert object store custom data history requests in regression algorithms
* Add custom object store data live data feed unit test
* Add multi-file object store custom data regression algorithms
* Minor fix
Calling UniverseDefinition.ETF("TICKER",...) in Python was resolving the UserDefinition.ETF(Symbol...) overload due to Symbol's implicit string operator, causing the symbol to be wrongly created.
* Changes up to the date
* Revert "Changes up to the date"
This reverts commit 45dc9756bc.
* First attempt to solve bug
* Improve solution
* Add unit tests
* Fix bugs and add regression tests
* Solve bugs
* Deprecates IAlgorithmSettings.DataSubscriptionLimit
We will let the brokerage check and notify users that they have subscribed to datafeeds beyond their quota.
* Removes Unit Test
Also remove usage in `EmptyEquityAndOptions400Benchmark`.
* Fix brokerage tests filled order modification race condition
* Listen to fill event in test setup in brokerage tests
* Minor changes
* Address peer review
* Minor changes
* Removed platformId support from C2.
Modernized C2 API calls to APIv4.
Added C2 rate limiters (RateGate).
Added response logging so users can easily debug their code.
Added a few symbols for the demo.
* Address requested changes
* Address requested changes
---------
Co-authored-by: Francis Gingras <francis@collective2.com>
* Add support for Combo Orders with Fill Model
- Override ComboMarketFill, ComboLimitFill, ComboLegLimitFill in
FillModelPythonWrapper.cs
- Modify FillModel.Fill() to use Python Fill model if it exists when the
order is COMBO
- Add regression algorithm ComboOrdersFillModelAlgorithm.py
* Add regression algorithm
* Add regression algos
* Fix bug
* Modify IB Fee model to work without order.Price
* Fix bugs and add unit tests
* Update regression tests
* Solve bug and address changes
* Nit change
* Fix stats
* Add unit test and test data
* Add unit test for model greeks accuracy and comparison
* Add regression test
* Correction on previous unit test to use realtime official IB Greeks
* Switch default American option price model to CRR model
* Address peer review
* Draft of unit test to reproduce error
* Fix bug and regression tests
* Remove draft unit test
* Check fees in ComboMarketOrderAlgorithm.cs
* Fix bug in fees for combo orders
- Update regression tests
- Solve bug in fees when combo orders had legs with different quantity
- Solve bug in fees when combo orders had legs with different security
types
- Update IB comission rate for options where the price is higher than 0.1 USD
* Add explaining comments
* Use leg security fee model
* Fix optimization candlestick handling
- Fixes for optimization candlestick handling
- Centralize deserialization of ChartPoint into new
ChartPointJsonConverter
* Minor fixes
- Update failing tests
- Update CI to fail if anything fails
* Candlestick charts base implementation
* Series and Candlestick series json serialization
* Some cleanup
* Add AddPlot method for candlestick series to QCAlgorithm
* Remove Values property from ISeriesPoint
* Add candlestick QCAlgorithm.Plot trade bar methods
* Implement candlestick series re-sampling
* Add more SeriesSampler unit tests
* Add examples of candlestick charts usage to exisiting charting algorithm
* Address peer review
* Address peer review
* Derive Candlestick from Bar
* Sampler changes
* Add new series types from the cloud
* Add more candlestick series sampler tests
* Minor cleanup
* Minor changes
* First attempt to fix the bug
* Allow the splitFactor to change over time
- Add unit test
* Nit change
* Address required changes
* Address required changes
* Adjust upper bound
* Solve bug in the parameters of the Report
Before, when the number of keys were odd a part of the raw template was
shown in the report generated. Additionally, the user couldn't define
the number of columns of the parameters table
* Add unit tests
* Enhance unit tests
* Nit change
* Add support for custom HTML file in Report
* Add support for custom crisis HTML template
* Revert "Add support for custom crisis HTML template"
After considering this option carefully, the QC team has decided to
add this functionality in a different way.
This reverts commit aead1ff850.
* Get parameters and crisis HTML custom templates
* Move parameters html template to a file
* Nit change
* Remove template.crisis.html and add unit tests
* Nit change
* Nit change
* Add unit tests for the fail cases
* Add trailing stop orders base implementation
* Handle trailing stop order prices rounding
* Implement trailing stop orders fill logic
* Minor fill model changes
* Add ApplySplit to fill model interface for models that might need to be aware of splits.
Filling trailing stop orders require keeping track of min/max prices, which need to be split adjusted.
* Add brokerage order updated event for communicating certain order types prices changes
* Add order update event args class for brokerage side order updates
* Revert IFillModel.ApplySplit
* Add trailing stop orders regression algorithm
* Updated order ticket demo algorithm to include trailing stop orders
* Some cleanup
* Support trailing stop orders in IB brokerage model
* Some cleanup
* Fix failing tests
* Fix failing regression algorithm
* Address peer review
* Add trailing stop price calculation unit tests
* Minor changes
* Minor change
* Add algorithm configurations to Report
* Add just Parameters section in the Report
* Revert "Add just Parameters section in the Report"
This reverts commit 775255127a212db81fc1cc1e23a35b7c60214eac.
* Include just AlgorithmConfiguration.Parameters
* Create ParametersReportElement
* Generate parameters into HTML dynamically
* Enable the user to modify CSS content throu
* Enable CSS override file to be passed through cmdline
* Check CSS Override file exists
* Fix bugs
* Address required changes
* Address required changes
* Address required changes
* Implement RangeConsolidator
It turned out that the behavior of RangeConsolidator was similar to
ClassicRenkoConsolidator. Then, some of the ClassicRenkoConsolidator
methods, were abstracted to new class called BaseTimelessConsolidator.cs,
from which both RangeConsolidator and ClassicRenkoConsolidator could
inherit, therefore resusing code. The following tasks were done:
- Create RangeConsolidator.cs
- Create RangeBar.cs
- Create BaseTimelessConsolidator.cs
- Create RangeConsolidatorTests.cs
- Modify ClassicRenkoConsolidator.cs
* Allow intermediate/Phantom RangeBar's
- Enhance unit tests
- Nit changes
- Allow intermediate/Phantom RangeBar's on RangeConsolidator
* Nit changes
* Create ClassicRangeConsolidator and more changes
- Add regression tests
- Enhance unit tests
* Address required changes
* Address requested changes
* Address requested changes
* Add regression tests with Tick Resolution
* Address required changes
* Increase Range for RangeConsolidatorWithTickAlgo
* Add more unit tests and solve bugs
* Nit change
- Minor adjustment not to transition from canceled to invalid order
status, which can happen in some cases in live trading as a race
condition. Adding unit test
- Make sure open order status is set to submitted if none provided,
adjusting existing test
- Always respect the minimum time in universe even if we haven't gotten
any data point yet. This is specially useful for live trading options
which are illiquid
* Changed argument exceptions so that they used the nameof() operator for parameter names
* Update IQSocket.cs
---------
Co-authored-by: John Doe <john@doe>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Contract chain selection improvements
- Adjust contract chain selection behavior for futures and options so we
are sure to use the correct local time when filtering expirations.
Adding unit test reproducing issue
* Update futures regression algorithms
* Address reviews
* Fix contract security filter universe
- Fix for contract security filter universe which would would use .Time
to filter expirations which could be yesterdays time at midnight
localtime. Adding unit tests reproducing issue
* Update regression algorithms
* Update MHDB CBOT Futures on Good Friday
* Update MHDB CME Futures on Good Friday
* Update MHDB CBOT Futures on Good Friday
* Update MHDB CME Futures on Good Friday
* Update MHDB comex and nymex futures on good friday
* Revert "Update MHDB comex and nymex futures on good friday"
Since all nymex and comex futures had a holiday on Good Friday, it's
better to add the holiday on `Future-nymex-[*]` and `Future-comex-[*]`
This reverts commit e111c8e461.
* Remove Good Friday holiday for cme and cbot futures
- Remove Good Friday holiday from `Future-cme-[*]` and `Future-cbot-[*]`
- Add Good Friday holiday for specific cme and cbot futures
* Add unit tests and update MHDB
- Add unit tests asserting LEAN reads correctly Good Friday holidays and
early closes for CME Group Futures
- Update the holidays for other CME group Futures on MHDB
* Fix bugs
* Enhance unit tests
* Add security dynamic properties unit and regression tests.
These assert that dynamic properties that are instances of python classes that inherit c# classes keep their python object references.
* Minor change
* Bump pythonnet version to 2.0.21
* Fix security dynamic properties cast from PyObject
* Minor change
* Minor tweaks to avoid CI tests random failures
- Increate timeout of research start test, using default value
- Minor tweaks for rate gate to improve performance
* Clean up rategate tests
* Update MHDB with NYSE and ICE holiday hours
- Update holidays and early closes for ´Equity-usa-[*]´
- Update holidays for certain products from Ice Futures US
* Add missing holiday hours in MHDB for CME-G Futures
- The missing holiday hours were from 3 July 2023
* Remove duplicates
* Address JB comment
* Remove repeated fields in BaseRenkoBar
- Add new unit test to all indicators that inherit from
IndicatorBase<TradeBar>. This test asserts the indicators can receive
RenkoBar's as input
* Add Regression test and improve unit tests
- Change `Volume` property from `RenkoBar.cs` since RenkoBar's don't
have Volume. Now it throws an error when accessed
* Address required changes
* Add Python regression test and enhance unit tests
* Enhance unit and regression tests
* Assert indicator current value at the end
* Allow more indicators to use Renko/VolumeRenkoBar
* Fix Security.IsMarketOpen for cases outside regular MH and last bar
* Minor changes
* Move bar check of IsMarketOpen into FillModel
* Minor changes
* Minor changes
* Minor changes
* Fix bug
- Add `ResetHandler()` and `IsReset` field in `CompositeIndicator.cs`
- Modify `CompositeIndicator.Reset()` to invoke the IsReset event
- Modify `VolumeWeightedAveragePriceIndicator.Reset()` method to reset
its composite indicator
- Modify `IndicatorExtensions.WeightedBy()` to register an event to the
returned composite indicator `IsReset` field
* Enhance implementation
* Address Requested changes
* Nit changes
* Nit change
* Nit change
* Address chages
- Make ResetCompositeIndicator inherit from CompositeIndicator and
override `Reset()` method
- Add ResetCompositeIndicator unit tests
- Modifiy IndicatorExtensions to keep returning the same type in
`WeightedBy()` method
* Avoid modify other IndicatorExtensions methods
* Nit changes
* Reset inner indicators and add more unit tests
* Dynamic security properties for quick objects storage
* Documentation and other minor changes
* Add Security indexer to access custom properties
* Address peer review
* Minor changes
* Improve regression algorithms
* Bumped pythonnet to version 2.0.19
* Bumped pythonnet to version 2.0.20
* Address peer review
* Fix bug and add unit tests
- Modify `InsightWeightingPortfolioConstructionModel.cs/py` to take the
absolute value in its `GetValue()` method
- Modifiy `InsightsReturnsTargetsConsistentWithDirection()` unit test to
consider also the case where the direction is Down and the weight is
negative
* Nit changes
* Fix bug
The algorithm `CustomPartialFillModelAlgorithm.cs` was not working as
expected with short orders because in its Fill model, the variable
`absoluteRemaining` instead of decrease was being increased since
the `FillQuantity` in each call to this method was negative. Therefore
when this amount was substracted to `absoluteRemaining`, instead of
substract, it was added. Hence the method created more partial orders
than expected, specifically: 580 short orders for SPY with Quantity -10.
* Fix `CustomPartialFillModelAlgorithm.py`
* Address required changes
* Nit change
* Address required changes
* Make statistics available at runtime to algorithms
* Re-calculate statistics on every call
* Housekeeping
* Add regression algorithms
* Address peer review
* Support for custom summary statistics at runtime
* Minor changes
* Address peer review
* Address peer review
* Minor changes
* Minor changes
* Fix bug and add unit tests
The bug was raised because rounding the first point inserted in the
RenkoConsolidator didn't get the closest y-multiple of the given number
x, where y is the bar-size. In order to compute the first y-multiple of
the given number x, we need to first compute x mod y := x - y*Floor(x/y),
once we get this number we divide it by y to know if we should round x
to minimum y-multiple greater than x, or should round x to the maximum
y-multiple smaller than x. If (x mod y)/y < 0.5 we round down, otherwise we round up (the amount (x mod y)/y is always less than one and higher than 0). See the Art of Computer Programming, Vol I. Donald E. Knuth.
- Add `GetClosestMultiple()` method in `RenkoConsolidator.cs`
- Add more unit tests
* Nit change
* Add XML docs
* Address required changes and add more unit tests
* Implement indicator history through a rolling window
* Minor changes and extend unit tests
* Add regression algorithms
* Minor changes
* Update regression algorithms examples
* Move resizing logic from IndicatorBase to RollingWindow
* Add unit tests and other minor changes
* Minor fix
* Add unit tests and other minor changes
* Add unit test
* Fix position group BP calculation
* Fix position group BP calculation
* Extend option startegy buying power unit tests
* Fix option strategy buying power calculation to account for order direction
* Fix bug, add unit tests and nit changes
The bug was raised because if two inputs points with repeated close values were inserted,
and the second one had 0 volume, when the first one is removed from
_oldDataPoints, as the reamaining value of it in _volumePerPrice is
zero, that entry is removed from _volumePerPrice. Therefore, when
the second input point is removed from _oldDataPoints, this entry
is not found in _volumePerPrice. Then the changes made were:
- Modify MarketProfile.Add() method to check every removed data point is
still in _volumePerPrice
- Add unit tests to covert this change
- Nit changes in VolumeProfileTests.cs
* Fix bugs and add more unit tests
- The Fill Forward bars shouldn't be considered in the calculations as
it's not real data.
- If bar inserted is not FF but has zero volume, it could remove a price
value from VolumePerPrice and leave this sorted list empty, then each
time we access that list we need to first check it's not empty
* Enhance ComputeNextValue() and add more unit tests
- Lean engine will automatically add an options underlying if not
present, but in most cases the option chain will select the underlying
too, so let's make sure the configurations match. Previous to this
change 'fill forward' setting could be different causing the
underlying to be duplicated in the data stack
- Adjusting live stack to emit live auxiliary events based on factor files.
Updating existing tests
- Minor improvements for map & factor file provider daily refresh time.
Adding unit tests
* Add ShortButterflyCall and ShortButterflyPut strategies helper factory methods
* Reduce duplication by adding the base OptionStrategyFactoryMethodsBaseAlgorithm algorithm class
* Housekeeping
* Fix bug in Beta statistics
- Beta in Report was calculated using equity points and not daily
performance points, as it's done in Backtest
- Performance points were not being divided by 100, as it's done in
StatisticsBuilder
- Benchmark points were not being converted into a percentage daily
performance of the benchmark, as it's done in StatisticsBuilder
* Improve implementation and add unit tests
* Enhance implementation and add more unit tests
* Enhance implementation
* Nit change
* Nit changes
* Make maintenance margin zero for long options
* Use OptionIntialMargin in OptionMarginModel
* Add naked call and put strategies unit tests
* Minor fix
* Account for premium in option strategies margin
* Minor fixes
* Add OptionInitialMargin class to separate strategies margin and premium
* Minor fix
* Minor fix
* Cleanup, docs and other minor changes
* Minor changes
* Minor changes
* Minor changes
* Address peer review
* Add SetAccountCurrency overload
- Add SetAccountCurrencyOverload to allow the user to set also account
currency quantity
- Add regression and unit tests to cover the changes
* Add regression algos
* Simplifiy SetAccountCurrency implementation
* Minor changes
* Minor changes
* Add support for Null SecurityPositionGroupModel
- Add support for algorithms to set the null security position group
model which will not group securities together but rather return the
single group buying power model. Adding regression algorithms
* Add Null Margin python regression & cleanup
- Add python regression algorithm & minor regression algorithms clean up
* Fix position group HasSufficientBuyingPowerForOrder
Comparing computed delta BP -- which already considers every impacted
group -- with the right available buying power (the remaining margin)
* Minor changes
* Pandas frame include all ticks
- Pandas data framde history response will include all ticks. Updating
existing and adding new tests
* Python pandas converter performance improvement
* Handle liquidating option position with an option strategy
* Handle liquidating option position with an option strategy
* Minor changes
* Minor fix in default margin model margin call order generation
* Add TriggerPrice field when applying a split
If one stock is split in k parts, the new TriggerPrice in an order for that stock should be also scaled by a factor of k/10. It was made:
- Modify `DefaultBrokerageModel.ApplySplit()` to adjust `TriggerPrice` field
- Regression and unit test was added
* Check Limit Price for LimitIfTouched
* Nit change
* MHDB will merge common entry
- The MHDB will merge the market and security common entry holidays,
early closes and late opens
* Normalize & reuse future US holidays
- Normalize & reuse future US holidays
* Update existing unit tests expected stats
* Fix PositionGroupBuyingPowerModel quantity for delta calculation
The model was calculation the order size that required a margin equal to
the given delta. This fix makes it so that it calculates the position
group order quantity required to get the given buying power delta,
taking into account the holdings for the existing position group in the portfolio.
* Fix PositionGroupBuyingPowerModel quantity for delta calculation
* Add unit tests
* Add unit tests
* Add unit tests
* Housekeeping
* Housekeeping
* Adapt unit tests and move messages to Messages class
* Improve quantity calculation for target BP loop
- PositionGroup.Quantity is now always positive.
- Added PositionGroupExtensions.Closes method and use it to check if reducing a position in PositionGroupBuyingPowerModel.GetPositionGroupBuyingPower. This is necessary since now a PositionGroup side is always "long" (see first bullet point).
* Improvements
* Improvements and new PositionGroup unit tests
* Unit tests improvements
- HasSufficientBuyingPowerForOrder test cases were completely updated to
make it accurate in terms of Lean's margin calculation.
- BuyingPowerForPositionGroupCalculation test was fixed in order to get
the correct expected result.
* Fixes for failing tests
* Added additional check to OptionStrategyMarginCallEventsAlgorithm
* Improve documentation comments
* Fix option strategies BuyingPowerForPositionGroupCalculation unit test
* Fix PositionGroupBuyingPowerModel.GetPositionGroupBuyingPower
Also address peer review
* Minor changes
* Update unit tests for new inverse strategies
Also fixed PositionGroupExtensions.WithQuantity to resolve inverse
strategy
* Minor fix
* Allow target/delta to go to the opposite direction of the position group
* Housekeeping
* Handle quantity for target/delta BP request for different reference position group
* Revert removal of PositionGroupBuyingPowerParameters.Direction
* Minor changes
* Peer review
* Minor tweaks on the position group buying power model
* Minor fix
* Implement Trailing FreePortfolioValue
- Implement Trailing FreePortfolioValue by default, users will be able
to set it to a fixed number if desired. Adding regression algorithm
- Setting the default 'MinimumOrderMarginPortfolioPercentage' from 0 to
0.1% of the TPV to avoud tiny trades by default
* Update existing regression algorithms
* Address reviews
- Send warning message to the user if a trade does not happen due to the
default setting of the minimum order margin percentage value
* Address reivews
* Rename TotalPortfolioValueLessFreeBuffer
* Update new regression algorithm
* Minor fix CrunchDao Symbology
- Minor fix for CrunchDao Symbology. Updating existing tests
* Add missing symbol mapping
- Add IAlgorithm.Ticker(Symbol) functionality which will return the
latest ticker for the requested symbol at the current algorithm time
- SignalExporters will use Ticker to get the current symbol ticker
* Rename SecId GetTicker to Ticker
* Solve bug and add regression test
The bug was raised because, when trying to use C#
MeanVarianceOptimizationPortfolioConstructionModel from a Python
algorithm, there wasn't a constructor that accepted a portfolio optimizer
as a PyObject. Additionally, there wasn't also a Python Wrapper to wrapp
that portfolio optimizer.
- Add PortfolioOptimizerPythonWrapper.cs
- Add constructor in
MeanVarianceOptimizationPortfolioConstructionModel.cs that accepts
portfolio optimizer as a PyObject
- Add regression algorithms to cover the changes
* Improve constructor overload implementation
* Change implementation to follow API pattern
* Enhance implementation and add unit tests
* Enhance implementation and add more unit tests
* Enhance implementation
* Implement Unregister helper method
- Implement Unregister helper method, each indicator which hold a
reference to the consolidators updating it. Adding regression
algorithms
* Fix for QuantBook indicators
* Address reviews
- Custom data universe selection market hours. Adding regression test
asserting the behavior. Updating existing tests due to market hours
change, triggering selection always, even the 4th of July 2018
* Add short call calendar spread strategy and review margin calculation
* Add short Put calendar spread strategy and fixed naked short options margin calculation
* Minor changes
* Minor changes
* Minor changes
* Solve the bug and add unit test
- The bug was raised because if something went wrong trying to register an indicator with a C# or a custom consolidator the exception raised was lost, because the catch sentence didn't return it with another exception
- A unit test was added asserting a exception with a related message was raised each time something went wrong with the consolidator
* Fix bugs and enhance unit tests
* Fix bug
* Handle unsupported orders and add unit tests
- Capture more Tradier Brokerage order limitations
- Add unit tests to cover this changes
* Enhance implementation and add more unit tests
* Enhance implementation and add unit test
* Remove IsMarketOrderGTC condition
* Remove error message MarketOrderIsGtc
* Covered and Protective Call and Put
- Improve covered call and put margin requirements based on IB
- Add missing protective call and puts strategies (inversion of a
covered call). Adding unit tests
* Fixes
* Add missing tests
* Expire Instead of Clear Insights In Base PCM In OnSecuritiesChanged
* Fixes Regression Test
The regression was incorrect because `_removedSymbols` didn't retain the removed symbols from previous `OnSecuritiesChanged` calls, and the algorithm relances once per month. E.g. On day 1, SPY and FB were removed, on day 3 AAPL and IBM were removed and `removedSymbols` would only include AAPL and IBM.
We should hve fixed this problem before with:
```csharp
if (_removedSymbols == null) _removedSymbols = new List<Symbol>();
_removedSymbols.AddRange(changes.RemovedSecurities.Select(x => x.Symbol));
```
However the change to use Expire fixes the issue.
* Removes Unused List of Removed Symbols
* Handle unsupported orders and add unit tests
- Capture more Tradier Brokerage order limitations
- Add unit tests to cover this changes
* Enhance implementation and add more unit tests
* Enhance implementation and add unit test
* Fix potential bug and add more docs
As FeeInBase and FeeInQuote are mutually exclusive, if one was set to true, the other one should be set to false. It was added information about where it's used and how fee currency selection works in Kraken.
* Nit change
* Simplify XML docs
* Fix the bug and add unit tests
- The bug was raised by different conditions.
First, the tick created in tick generator for Open Interest was not of type Open Interest.
Second, TickAggregator.cs was made to use daily resolution for OpenInterest always.
Third, the RandomValueGenerator, generated a random friday as expiration date for the option created but this one needed first two underlying data points, so if the expiration date was before the start date plus 3 days, RandomDataGenerator just generated OpenInterest data for the option.
* Add docs
* Add option strategy position group BPM tests
Tests added for calculating order quantity for a delta buying power
* Add option strategy position group BPM margin calculation tests
* Add option strategy position group BPM margin calculation tests
* Add option strategy position group BPM margin calculation tests
* Add option strategy position group available BPM calculation tests
* Add option strategy position group impact buying power calculation
* Housekeeping
* Unit tests improvements
* Unit tests improvements
* Add more unit tests
* Address peer review
* Add forward data only fill model example
- Add forward data only custom fill model C# & PY example.
- Minor adjustment for 'MarketOrderFillTimeout' to be zero always in
backtesting
* Address reviews
* Add support for brokerage side new orders
- Add support for brokerage side new order events for liquidation cases
* Minor cash delta fix
* Improve account cash logging
* Fix null reference exception for open orders
* Implement ShortableProviderPythonWrapper.cs
- Modify AllShortableSymbolsCoarseSelectionRegressionAlgorithm.cs and ShortableProviderOrdersRejectedRegressionAlgorithm.cs to use ShortableProvider from Security and not from the Brokerage model
- Add SetShortableProvider() overload method in Security.cs to set a custom shortable provider from Python
- Remove AllShortableSymbols() method from LocalDiskShortableProvider.cs
- Remove DefaultShortableProvider class
- Add regresion algorithms in C# to cover the changes done
* Revert "Merge process"
This reverts commit 775a4b8ec1, reversing
changes made to bcc3e790f6.
* Revert "Revert "Merge process""
This reverts commit aa18fb40ee.
* Solve bugs
- Solve bugs in DefaultMarkets, CanUpdateOrder() and CanSubmitOrder() methods from BrokerageModelPythonWrapper.cs
- Add regression algos in both Python and C# to cover the changes
* Catch potential errors and add unit tests
* Add more unit tests
* Nit change
* Enhance implementation, add more unit tests
* Add IndexOptionPriceVariationModel
- Add IndexOption price variation model to handle price based minimum
price variation. Adding unit tests
* Minor improvements for VIX & VIXW
* Refactor settlement mode api
- Add new Scan method on ISettlementModel API. To be used by
futures settlement model mark to market handling
- Use 'Parameters' classes for 'ISettlementModel'
* Address reviews
* Daily future settlement
- Implement daily future settlement. Adding unit & regression tests
* Address reviews
* Assert cash balance every day
* Refactor settlement mode api
- Add new Scan method on ISettlementModel API. To be used by
futures settlement model mark to market handling
- Use 'Parameters' classes for 'ISettlementModel'
* Address reviews
* Fix multiple option strategies liquidation
When holding multiple position groups of the same underlying, reverting
one of the groups was not possible because of the presence of the other
position groups, which caused no matching groups to be formed for the
order.
* Make OptionStrategyPositionGroupResolver not group single position
* Revert make OptionStrategyPositionGroupResolver not group single position
* Implement ShortableProviderPythonWrapper.cs
- Modify AllShortableSymbolsCoarseSelectionRegressionAlgorithm.cs and ShortableProviderOrdersRejectedRegressionAlgorithm.cs to use ShortableProvider from Security and not from the Brokerage model
- Add SetShortableProvider() overload method in Security.cs to set a custom shortable provider from Python
- Remove AllShortableSymbols() method from LocalDiskShortableProvider.cs
- Remove DefaultShortableProvider class
- Add regresion algorithms in C# to cover the changes done
* Revert "Merge process"
This reverts commit 775a4b8ec1, reversing
changes made to bcc3e790f6.
* Revert "Revert "Merge process""
This reverts commit aa18fb40ee.
* Solve bug
- Add new constructor overload in OpenInterestFutureUniverseSelectionModel.cs that accepts future chain symbol selector as PyObject
- Add a private static method in OpenInterestFutureUniverseSelectionModel that converts Python lambda function to Func<DateTime, IEnumerable<Symbol>>
- Add a regression algorithm in Python to cover changes. In these case, add Python version of OpenInterestFuturesRegressionAlgorithm.cs
* Nit changes
* Nit change
* Minor docs tweak
---------
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Implement ShortableProviderPythonWrapper.cs
- Modify AllShortableSymbolsCoarseSelectionRegressionAlgorithm.cs and ShortableProviderOrdersRejectedRegressionAlgorithm.cs to use ShortableProvider from Security and not from the Brokerage model
- Add SetShortableProvider() overload method in Security.cs to set a custom shortable provider from Python
- Remove AllShortableSymbols() method from LocalDiskShortableProvider.cs
- Remove DefaultShortableProvider class
- Add regresion algorithms in C# to cover the changes done
* Implement ShortableProviderPythonWrapper.cs
- Modify AllShortableSymbolsCoarseSelectionRegressionAlgorithm.cs and ShortableProviderOrdersRejectedRegressionAlgorithm.cs to use ShortableProvider from Security and not from the Brokerage model
- Add SetShortableProvider() overload method in Security.cs to set a custom shortable provider from Python
- Remove AllShortableSymbols() method from LocalDiskShortableProvider.cs
- Remove DefaultShortableProvider class
- Add regresion algorithms in C# to cover the changes done
* Solve bugs and nit change
* Address review
---------
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Solve Report Generation Conversion Rate Error bug (#7183)
* Solve bug
Remove if sentence that checked if Portfolio contained securities that
needed a conversion, like Forex, Crypto or Cfd. It didn't take into
account CryptoFutures so when the backtest result used that kind of
security types it never found a conversion for them. Making a git blame
for Report.PortfolioLooper constructor it was found the if sentence was
not needed so it was removed and then different reports were made for
algorithms that used security types different than Forex,Crypto,Cfd and
CryptoFutures to check everything was still working as expected.
* Solve bug
Remove if sentence that checked if Portfolio contained securities that
needed a conversion, like Forex, Crypto or Cfd. It didn't take into
account CryptoFutures so when the backtest result used that kind of
security types it never found a conversion for them. Making a git blame
for Report.PortfolioLooper constructor it was found the if sentence was
not needed so it was removed and then different reports were made for
algorithms that used security types different than Forex,Crypto,Cfd and
CryptoFutures to check everything was still working as expected.
* Missing Python Version of Crypto Future Algorithms - Issue 7128
* Address reviews
---------
Co-authored-by: hungv <hungv@compatibl.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add dataMappingMode parameter to every history api method overload
* Minor unit tests fixes
* Update regression algorithm stats
* Minor changes
* Minor changes
- Validate cache folder exists before using it
- Avoid duplication fetching factor & map files path
- Adding helper method to check if directory is empty. Adding unit tests
- Create `SecurityDataFilterPythonWrapper.cs`
- Modifiy `Security.cs` to set a data filter, whether is a C#, Python derivative or custom data filter custom
- Add Python and C# regression algorithms to assert the changes
* Add extendedMarket parameter to every history api method overload
* Rename extendedMarketHours parameter
New name is extendedHours as in the History API to standarize parameters naming
* Update generic history overloads to use every matching subscription
* Update regression algorithms stats
* Centralize period-based history error for tick resolution
* Rename extended market hours parameter to extendedMarketHours
* Minor changes
* Minor changes
* Minor unit tests changes
* Minor unit tests changes
* Minor changes
* Minor unit tests changes
* Minor unit tests changes
* Added PyStan Library in DockerfileLeanFoundation
* Add new libraries and unit tests
---------
Co-authored-by: Ivan Radigales Creus <ivanrdgc@gmail.com>
* Implement `OptionExerciseModelPythonWrapper.cs`
- Modifiy `Option.cs` to set the automatic exercise model, even if it's a python custom option model exercise model
- Add `QuantConnect.Orders.OptionExercise` in `AlgorithmImports.py`
- Add regression algorithms in C# and Python showing how to specify a custom option exercise model
* Address required changes
- Remove unnecessary constructor in
´CustomOptionExerciseModelRegressionAlgorithm.cs´
- Change `.Dispose()` in `OptionExerciseModelPythonWrapper.OptionExercise(9` to ´using´ keyword
- Change `Option.SetOptionExerciseModel()` XML docs
* Minor tweaks
* Update CustomOptionExerciseModelRegressionAlgorithm.cs
---------
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add fillForward parameters to every History method
* Remove conflicting Python history method
* Undo removing conflicting Python history method
* Minor changes
* Minor changes
* Minor changes
* Add fillForward and extendedMarket parameters to history request factory
* Minor changes
* Minor changes
* Minor changes
* Minor changes
* Minor unit tests changes
* Rename fillForward parameter in History API
New name if fillDataForward as in the Add*Security API to standarize
parameters naming
* Rename fillForward parameter
Using the shorter fillForward in every API
* Solve bug
Remove if sentence that checked if Portfolio contained securities that
needed a conversion, like Forex, Crypto or Cfd. It didn't take into
account CryptoFutures so when the backtest result used that kind of
security types it never found a conversion for them. Making a git blame
for Report.PortfolioLooper constructor it was found the if sentence was
not needed so it was removed and then different reports were made for
algorithms that used security types different than Forex,Crypto,Cfd and
CryptoFutures to check everything was still working as expected.
* Solve bug
Remove if sentence that checked if Portfolio contained securities that
needed a conversion, like Forex, Crypto or Cfd. It didn't take into
account CryptoFutures so when the backtest result used that kind of
security types it never found a conversion for them. Making a git blame
for Report.PortfolioLooper constructor it was found the if sentence was
not needed so it was removed and then different reports were made for
algorithms that used security types different than Forex,Crypto,Cfd and
CryptoFutures to check everything was still working as expected.
* Add Collective2SignalExportClass
Add SignalExportTarget interface
* Collective2SignalExport test working
Add SignalExportManager
Add draft of CrunchDAOSignalExport
* Modify SignalExportManager
Instantiate SignalExportManager in QCAlgorithm constructor
Draft of CrunchDAOSignalExport
* Improve SignalExportManager
- Add regression tests SignalExportDemonstrationAlgorithm in C# and
Python
* Improve SignalExportDemonstrationAlgorithm
Address requested changes in Collective2SignalExport, SignalExportManger and SignalExportTargetTests.cs
* Add CrunchDAOSignalExport.cs
Add CrunchDAOSignalExport unit tests in SignalExportTargetTests.cs
* Add NumeraiSignalExport.cs
Modify SignalExportDemonstrationAlgorithm.cs to test NumeraiSignalExport
Add unit test in SignalExportTargetTests to test NumeraiSignalExport
* Address required changes
- Add BaseSignalExport.cs
- Add SignalExportParameters.cs
- Add PortfolioSignalExportDemonstrationAlgorithm.cs/py
- Improve Error handling in SignalExport provider classes
- Collective2SignalExport now gets the correct number of shares for each holding
- SignalExportManager now computes the correct holding percentage of each portfolio target
- SignalExportManager now takes into account if the algorithm is in live mode
- Demonstration algorithms now are more simple
* Address last required changes
- PortfolioSignalExportDemonstrationAlgorithm.cs/py now inherits from SignalExportDemonstrationAlgorithm.cs/py
- Add more unit tests to assert SignalExportManager gets the correct percentage quantity for each holding
- Change Collective2SignalExport, CrunchDAOSignalExport and NumeraiSignalExport Send() method to return true if there was no error while sending the signals and false otherwise
- Nit changes
* Remove exceptions thrown
- Add more unit tests and more test cases
- Enhance BaseSignalExport.Dispose() method
* Add Collective2SignalExportClass
Add SignalExportTarget interface
* Collective2SignalExport test working
Add SignalExportManager
Add draft of CrunchDAOSignalExport
* Modify SignalExportManager
Instantiate SignalExportManager in QCAlgorithm constructor
Draft of CrunchDAOSignalExport
* Improve SignalExportManager
- Add regression tests SignalExportDemonstrationAlgorithm in C# and
Python
* Improve SignalExportDemonstrationAlgorithm
Address requested changes in Collective2SignalExport, SignalExportManger and SignalExportTargetTests.cs
* Add CrunchDAOSignalExport.cs
Add CrunchDAOSignalExport unit tests in SignalExportTargetTests.cs
* Add NumeraiSignalExport.cs
Modify SignalExportDemonstrationAlgorithm.cs to test NumeraiSignalExport
Add unit test in SignalExportTargetTests to test NumeraiSignalExport
* Address required changes
- Add BaseSignalExport.cs
- Add SignalExportParameters.cs
- Add PortfolioSignalExportDemonstrationAlgorithm.cs/py
- Improve Error handling in SignalExport provider classes
- Collective2SignalExport now gets the correct number of shares for each holding
- SignalExportManager now computes the correct holding percentage of each portfolio target
- SignalExportManager now takes into account if the algorithm is in live mode
- Demonstration algorithms now are more simple
* Address last required changes
- PortfolioSignalExportDemonstrationAlgorithm.cs/py now inherits from SignalExportDemonstrationAlgorithm.cs/py
- Add more unit tests to assert SignalExportManager gets the correct percentage quantity for each holding
- Change Collective2SignalExport, CrunchDAOSignalExport and NumeraiSignalExport Send() method to return true if there was no error while sending the signals and false otherwise
- Nit changes
* Remove exceptions thrown
- Add more unit tests and more test cases
- Enhance BaseSignalExport.Dispose() method
* Fix failing regression tests
* Fix failing unit tests
* Nit changes
* Nit change
* Nit change
* Fix failing unit tests
* Changes required
- Break regression algos `SignalExportDemonstrationAlgorithm.cs/py` nad `PortfolioSignalExportDemonstrationAlgorithm.cs/`y` into three ones, one for each signal export provider
- Change SignalExportManager constructor to receive current algorithm as a parameter
- Fix bug in `SignalExportManager.GetPortfolioTargets()`, now it computes the correct percentage for each holding
- Make `BaseSignalExport.DefaultAllowedSecurityTypes` overrdible
- Handle case were `Collective2SignalExport.ConvertPercentageToQuantity()` returns null
- Clean unnecessary code in `Collective2SignalExport()`, `CrunchDAOSignalExport()` and `NumeraiSignalExport()`
* Nit change
* Nit change
* Minor tweaks after review
* Remove indexes from signal exports
* Required changes
- Change EMA indicators period from 200, 300 to 10,100 in regression algorithms
- Remove Indices from regression algorithms
- Add more XML documentation to regression algorithms
- Change `Log.Error` to `_algorithm.Error` in Signal export providers. Besides, fix error message format
- Change default value for `platformId` parameter in `Collective2SignalExport.cs` constructor
- Solve small bugs in SignalExportProvider when verificating the amount of porfolio targets is greater than zero and each portfolio target is allowed
- Handle case when `PortfolioTarget.Percent()` returns null in `Collective2SignalExport.ConvertPercentageToQuantity()`
- Handle error format message from Collective2 API
- Check every ticker signal is between 0 and 1 (inclusive) in `CrunchDAOSignalExport.cs`
- Modifiy `NumeraiSignalExport.cs` constructor to take into account filename given in the arguments
- Fix small bug with the return value of `ConvertTargetsToNumerai()` method in `NumeraiSignalExport.cs`
- Modify `SignalExportManager.cs` to return true when the algorithm being ran is not in live mode
- Remove indices from CrunchDAO unit tests
* Enhance ´CrunchDAOSignalExport.cs´ implementation
---------
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix EMA indicator first value calculation
As done by TALib and TradingView, the first EMA value after warmup is
the SMA of the first period.
* Update TSI indicator test data
Data was exported from TradingView
* Update TRIX indicator test data from TradingView
* Update AccumulationDistributionOscillator indicator test data from TradingView
* Update Double EMA indicator test data from TradingView
* Update McClellanSummationIndex indicator test data
* Update SchaffTrendCycle indicator test data
* Update TripleExponentialMovingAverage indicator test data from TradingView
* Update stats for algorithms using EMA
* Update failing unit tests
* Handle position group margin calls
* Minor changes
* Update regression algorithms
* Minor changes
* Minor changes
* Added regression algorithm
* Peer review
* Peer review
* Peer review
* Peer review
* Minor changes
* Minor changes
* Add unit test
* Allow sufficient buying power when closing position group
* Add unit test
* Improve regression algorithms
* Fix ReportCharts color_map bug
- Add ´CryptoFuture´ entry in `color_map` dict using color #E55812
- Make ´ReportCharts.py´ color_map a class attribute
- Change ´ReportChartsTests.cs´ to use color_map dict from ´ReportCharts.py´
- Add unit test that fails every time a new SecurityType is added
* Improve unit test
* Nit change
* Nit change
* Updated MHDB and CME currencies futures expiry functions
Some expiry functions needed to take into consideration the security
holidays
* Add more holidays to CME securities in MHDB
* Update regression test to check number of insights
* Update model to have insight manager ref and cancel insight when signal goes flat
* Update unit tests and regression tests
* Address peer review: using Insights.Clear
* Invalidate option strategies orders when short selling over margin
When shorting an option strategy with margin requirements that cannot be
met, the order should be invalidated. The buying power model is now able
to detect said situation and result in unssuficient buying power.
The OptionsMarginModel now uses the parameters instance values instead
of the security holdings to compute margin requirements. This fixes the
situation when calculation buying power for a first time position with
no holdings.
* Add and fix unit tests
* Update regression algorithms
* Peer review
* Downloader data provider fixes
- Fix stored data time zone, which is expected in data time zone
- Minor logging improvements
- LeanData Parse handles etf universes path. Adding unit test
- LeanDataWrite will only map if there's something to map to
* Fix download data provider synchronization
* Synchronization improvements after more testing
- KeyStringSynchronizer will handle reentrancy. Adding unit tests
- Define a unique Synchronizare for downloading data, since not related
to disk reading & writting
* LeanDataWriter can handle mapping
- Adjust LeanDataWrite so that it can handle symbol mapping if required.
Adding unit tests
* Fix Lean Data path parsing
* Updates Regression Algorithm to Assert the Number of Insights
* Updates HistoricalReturnsAlphaModel to Cancel Insights Not Emit Flat
We expect fewer insights after this change, but no changes to any regression algorithm.
* Updates Regression Algorithm
Assert the new expected number of generated insights.
* Use InsightCollection Clear Method
Replaces old `TrailingStopRiskFrameworkAlgorithm` for `TrailingStopRiskFrameworkRegressionAlgorithm` that inherits from `BaseFrameworkRegressionAlgorithm` as a member of the framework regression tests.
* Update regression test
* Update model to expire insights
* Avoid remove insights
* Update regression test
* Use InsightManager Cancel to Expire All Insights
* Update CompositeRiskManagementModelFrameworkAlgorithm
This regression also depends on `MaximumUnrealizedProfitPercentPerSecurity` and `MaximumDrawdownPercentPerSecurity` but `MaximumDrawdownPercentPerSecurity` doesn't close positions.
---------
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
* Remove OnEndOfAlgorithm and update expected trades
* Update models to cancel insights
* Update expected results
There are 3 trades instead of 2 because the PCM does a rebalance
* Remove `Remove` method call
* Update to use the new `Cancel` method
If we use `.astimezone(dt.tzinfo)`, the new time was convering the timezone from local (e.g. PST) to GMT (dt.tzinfo). In this case, we only want to remove the timezone to enable the operation.
* Renames and Updates BaseAlphaModelFrameworkRegressionAlgorithm
The `BaseFrameworkRegressionAlgorithm ` will be used for multiple framework regression tests
* Updates and Renames EmaCrossAlphaModelFrameworkAlgorithm
Adds "Regression" to inform that it's a regression algorithm.
* Updates and Renames MaximumPortfolioDrawdownFrameworkAlgorithm
Adds "Regression" to inform that it's a regression algorithm, and use the model name: `MaximumDrawdownPercentPortfolio`
* Adds New Regression Algorithms
* Adds BaseAlphaModelFrameworkRegressionAlgorithm
`BaseAlphaModelFrameworkRegressionAlgorithm` will be used to validate Alpha Model regression algorithm with the same universe.
- HistoricalReturnsAlphaModelFrameworkAlgorithm
- EmaCrossAlphaModelFrameworkAlgorithm
- MacdAlphaModelFrameworkAlgorithm
- RsiAlphaModelFrameworkAlgorithm
- BasePairsTradingAlphaModelFrameworkAlgorithm
* Addresses Peer-Review
* Fix fill quantity using group order quantity for combo orders
* Add Order's ComboQuantity property
* Add Order's ComboDirection property
* Minor changes and regression algorithms update
* Minor changes
* Update algorithms stats
* Minor changes and regression algorithms update
* Store the full quantity for each combo order leg in Quantity property instead of the ratio
* Minor changes and regression algorithms update
* A few fixes after pair programming
* Handle grouping position reduction
---------
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Improves MinimumVariancePortfolioOptimizer Values Handling
The C# version of the `MinimumVariancePortfolioOptimizer` generated NaN resulting in unit tests failing.
If the solver returns NaN for an item, we set it to zero. if all items are NaN or Zero, we return the initial guess, since the sum cannot be zero.
* Fixes `RiskParityPortfolioConstructionModelTests`
We need to add insights to insight manager before we call `CreateTargets`.
* Adds Unit Test
Asserts that we can remove a consolidator that was previously added.
* AddConsolidator Method Only Wraps Non-C# Consolidators
This behavior was implemented in `QCAlgorithm.RegisterIndicator` to avoid wrapping an object unnecessarily.
- Improve LeanData aggregation helper methods to use shared consolidator based implementation
- Adding Tick to tradeBar aggregation
- Adding more unit tests
- Fix data-queue-handler configuration initialization using paper brokerage
- Fix for downloader data provider request times
- Fix null reference exception in live result handler if algorithm is
null
* Asserts Number of Insights In One PCM Regression Test
If the `EqualWeightingPortfolioConstructionModel` interacts with the `QCAlgorithm.Insights`, the number of elements in the collection should not the sum of emitted insights.
* Refactor Portfolio Construction Models to Use Insight Manager
`PortfolioConstructionModel` will use `QCAlgorithm.Insights" instead of class property `InsightCollection` to manage the insights. It no longer adds insights to the collection, but it removes them if they expire or the securities are removed from the universe.
Updates PCMs that were affected by the change.
* Updates Unit Tests
We need to add the insights to the insight manager before we call `PortfolioConstruction.CreateTargets`
* Minor IndexOption underlying TZ fix
- Fix for option universe chain underlying TZ which might not match the
options one, bug affected IndexOption specifically.
* Update affected regression tests
* Performance Improvements
- Remove Immutable collection from position groups to improve
performance. Adding more tests
- Remove ConcurrentDictionary from PortfolioTargetCollection &
DataManager. Adding more tests
- Add Securities enumerator keys & values cache
* Add test for PortfolioTargetCollection remove by reference
* Apply splits and dividends to volatility models
* Apply splits and dividends to volatility models using history requests
* Add new ScaleRaw data normalization mode
Handling the new mode in the price scale enumerator.
* DataNormalizationMode.ScaledRaw history requests
* Minor changes
* Minor changes
* Disable new normalization mode in AddSecurity methods and other minor changes
* Peer review
* Minor changes
* Peer review
* Minor changes
* Peer review
* Peer review
* Peer review
* Add scaled raw history regression algorithm
* Add more regression algorithms
* Add more regression algorithms
* Add Slice.TryGet unit tests
* Peer review
* Peer review
* Peer review
* Peer review
* Peer review
* Update algorithms stats
* Peer review
* Peer review
* Adds Unit Test For Limit Price Above/Below Open
See `LimitOrderFillsAtOpenWithFavorableGap`
* Fixes Limit Price Above/Below Open
If we place a buy/sell limit order below/above the current market price in TWS it fills immediately, so we model this behavior by filling with the opening price of the first trade bar.
* Updates Regression Tests
The number of trades did not change as expected. The fills are better because orders are filling with the open price when the new condition is met.
* Refactor alpha statistics
- Refactor alpha statistics, cleaning up and simplifying no longer required calculations and scoring
- Adding new InsightEvaluator abstraction, adding C# & PY regression
algorithms
* Optimization backtest result json converter update
* Address reviews
- Remove IAlphaHandler, move insight storage responsability to IResultHandler
and centralizing insight collection on the QCAlgorithm.Insights to be
reused by the framework models
- Fix portfolio turnover single day backtests and duplicate time
sampling handling. Updating regression algorithms
* Add InsightCollection tests and minor fixes
* Adding more & improved tests
The `LimitFill`, `LimitIfTouchedFill` and `StopMaketFill` methods require `TradeBar` data to test price levels. We create a method to serve these methods as well other methods that will require this information.
Removes `Price > 0` check for `TradeType.Trade` data. Unecessary check, since a tick price cannot be less or equal to zero.
* Adds Unit Tests for Stop Market Fill
These unit tests covers new scenarios:
- Cannot trigger on quote data (QuoteBar or TickRype.Quote);
- Adds missing test for tick data (TickType.Trade)
- Unfavorable gap (see QuantConnect#4545)
* Fixes EquityFillModel StopMarketFill
Use the entire tick history to find the trigger price
Fill price is the stop price, unless there is an unfavorable gap (see #4545).
* Update Regression Tests
All regression tests keep the total trades. The difference are due to the time that the stop market orders are triggered and the prices that are filled. The `StopLossOnOrderEventRegressionAlgorithm` and the `UpdateOrderRegressionAlgorithm` observe the unfavorable gap case.
* Addresses Peer-Review
* Updates Regression Test
New `OrderListHash` because of the change in the fill on open case message.
* remove annualization
* Update python optimizer to best fit in convex problem
* rerun test
* test metric
* Add comment to explain using simple return
* Add unit test on PR
* Removes PythonTools from Solution
Missing removal from #6967.
It results in a runtime error if we use dotnet to list the packages:
`dotnet list ../Lean/QuantConnect.Lean.sln package`
* Update QuantConnect.Lean.sln
---------
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Foundation Update
* Adding more foundation packages
* Update existing libraries
* Further package updates
* Image cleanup
- Update base image to ubuntu 22.04
- Some clean up on dockerfile
- Remove ARM base image
* Test updates
- Minor docker file tweaks
- Update existing python environment tests
* Add more tests & fix Julia
* Remove jupyter-dash
* Remove R from foundation
* Remove Julia
* Remove Tensorforce
* Adds Regression Test
The new regression test asserts that the total trades is 2.
* Fixes Liquidate Existing Holdings Bug
Liquidate existing holdings before open new postions.
* Addresses Peer-Review
* IsOption uses underlyings holidays
- Option assets will use underlyings holidays, early closes and late
opens if their own are not defined
* Update regression algorithms
* Adds Unit Tests
The `LimitFill` method should not fill using `QuoteBar` or `Tick` with `TickType.Quote` type.
Adds missing test for tick data (`TickType.Trade`).
* Fixes EquityFillModel.FillModel
Use `Tick` with `TickType.Trade` or `TradeBar` information to fill limit orders.
* Update Regression Test Expected Statistics
The regression tests changed because of different fills.
The `ExtendedMarketTradingRegressionAlgorithm` has different number of trades because of an extra fill on the 4th order generated by TradeBar with a Low lower than than the QuoteBar.Ask Low:
> 20230222 13:56:24.251 TRACE:: Log: Time: 10/10/2013 12:01:00 OrderID: 4 EventID: 2 Symbol: SPY Status: Filled Quantity: 10 FillQuantity: 10 FillPrice: 143.8998 USD
> asset.Cache.GetData<QuoteBar>().ToString()
"SPY: Bid: O: 144.2457 Bid: H: 144.2629 Bid: **L: 144.2457** Bid: C: 144.2629 Ask: O: 144.2543 Ask: H: 144.2889 Ask: **L: 144.2543** Ask: C: 144.2889 "
> asset.Cache.GetData<TradeBar>().ToString()
"SPY: O: 144.2543 H: 144.4532 **L: 143.4156** C: 144.2716 V: 75423"
* Improves Tick Resolution Unit Test
* Fixes Tick Resolution Case Handling
`master` only considers the latest trade, missing possible fills in the batch of trades.
* Adds Unit Test for Gap
See https://github.com/QuantConnect/Lean/issues/963
* Addresses Fill Optimistic Assumption
If we have a bar that gaps in our favor, we accept the limit price to avoid optimitic fills.
* Fixes Regression Tests
All regression tests with limit orders have worst performance after we remove the optimitic assumption, and use the limit price instead.
* Updates the PearsonCorrelationPairsTradingAlphaModelFrameworkAlgorithm
Change the `PearsonCorrelationPairsTradingAlphaModelFrameworkAlgorithm` logic to show that it doesn't remove the consolidators used in the Alpha Model's indicators.
* Fixes `BasePairsTradingAlphaModel`
The `BasePairsTradingAlphaModel` will create indicators with class constructors and register them to consolidators that will be removed when the security is removed from the universe.
* Addresses Peer-Review
* Update Oanda's Forex and Metals CFDs trading hours in the MHDB
* Fix failing unit tests
* Update regression algorithm statistics
* Fix failing unit tests
* Minor changes
* Fix Issue #6937
Some brokerage models don't check the order type in `CanSubmitOrder`.
As a result, unimplemented order types don't cause `CanSubmitOrder`
to return false as it should.
This commit fixes these issues for:
* `FTXBrokerageModel.cs`
* `InteractiveBrokersBrokerageModel.cs`
* `SamcoBrokerageModel.cs`
* `TradierBrokerageModel.cs`
* `ZerodhaBrokerageModel.cs`
Brokerage implementations can be found at
https://github.com/orgs/QuantConnect/repositories. For example, the
Zerodha Brokerage implementation is available at
https://github.com/QuantConnect/Lean.Brokerages.Zerodha.
Signed-off-by: Abhishek Kumar <abhi.kr.2100@gmail.com>
* Use HashSet instead of an array
Checking if an object is contained within a HashSet is faster than
checking the same in an array.
Signed-off-by: Abhishek Kumar <abhi.kr.2100@gmail.com>
* Minor adjustments and normalization
* Add missing OptionExercise for IB
---------
Signed-off-by: Abhishek Kumar <abhi.kr.2100@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix live trading chart series index
- Fix Series loosing their index when handled by the
LiveTradingResultHandler
- Some clean up for Series.cs
* Fix SeriesSampler loosing Series settings
* Adds NullBuyingPowerModel
The `NullBuyingPowerModel` considers that we have sufficient buying power for all orders.
Adds example using a bull call spread since a equity buy and hold would not show the impact of this model in the position group buing power model.
* Define NullBuyingPowerModel.GetMaintenanceMargin
The `NullBuyingPowerModel.GetMaintenanceMargin` returns a `MaintenanceMargin` of zero. It means the total margin used is always zero and margin calls will not be triggered.
This feature is inspired on the `ConstantBuyingPowerModel`. We don't inherit from that class, because `ConstantBuyingPowerModel.GetInitialMarginRequirement` gives us very small initial margin that leads to a very large quantity if we use `SetHoldings` or `CalculateOrderQuantity`
The calculations of `CloseTimeUtc` are correct and consistent: if the exchange is closed when the insight expires, `CloseTimeUtc` is the next open.
However, the definition of the `Period` was not consistent. Normally, it is defined by the different between `CloseTimeUtc` and `GeneratedTimeUtc`, but when we create an Insight with the Resolution and bar count overload, the period was defined by them:
```csharp
insight.Period = Resolution.ToTimeSpan().Multiply(BarCount);
```
which is incorrect, since it doesn't take the days the exchange is closed into account. The `ComputePeriod` method enforced consistency between the Resolution and bar Count overload and the `DateTime` overload, so the period for this overload was also incorrect.
* Custom data type history request in python
* Potential solutions
* Minor changes
* Use Slice.Get(Type) for getting python custom data history
* Minor changes
* Add unit tests
* Add unit tests
* Udpdate regression algorithms
* Peer review
* Peer review
* Add research regression tests
* Minor changes
* Minor changes
* Minor tweaks
---------
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix index option data timezone
- Fix index option data timezone which was 1 hour late than expected
* Update existing index option tests
- Update existing index option tests.
- Disable index option daily resolution support
* Fix future daily price scale factor update
- Fix future daily price scale factor update which was happening 1 day
late due to data and exchange time zone differences.
Adding regression algorithm reproducing and asserting it.
* Address review
* Update unsettled cashbook cash conversion rates
Also moved conversion rate holding from Cash to the CurrencyConversion
instance
* Minor fixes
* Peer review
* Minor fixes
* Minor fixes
* Minor fixes
* add wkhtmltopdf to lean image
* convert html to pdf
* avoid black stripts at bottom of each page
* address reviews
* address review
* address reviews
* self review
* address review
* Fix security cache reset
- Fix security cache reset so it only happens when no universe has it as
a member. Adding regression algorithm reproducing issue
* Avoid non deterministic FillForward behavior
* Fix default future chain universe selection
- Fix default future chain universe selection that should always return
empty, no selection. In some cases it would allos some contracts
though. Adding regression algorithm reproducing issue.
* Address review
* Minor debugging improvements
- Improve exception message if user provides empty/null ticker
- Log margin information on algorithn initialization
- Add unit test asserting option strategy position groups resolve
correctly for existing holdings
* Symbol Lazy SecurityDefinitionSymbolResolver
- Symbol will have a lazy SecurityDefinitionSymbolResolver instance
resolution
* Use any resolution for chain provider
- Use any data resolution available to source symbols for the file based
chain provider. Adding unit test
* Fix selection timezone bug
- Fix universe selection timezone bug. Updating regression algorithms
* Handles Data Subscriptions with One Security Type
Polygon sells data subscriptions for security types separately, so we should not test whether the users have all three subscriptions, Equity, Forex, and Crypto. Users just need the type they will use.
Adds new crypto markets, and unit tests for them. Adapt unit test.
* Fixes ToolBox Downloader
Adds `--api-key` argument.
Closes#6687
* Address reviews
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Expand Index option support
- Adjust APIs so that the same underlying can be used
for different options, adding support for SPX weekly options. Adding
regression tests
* Fix IndexOption.IsStandard
* Add IndexOption test
Fixed this error from happening when running this code:
tf.placeholder() is not compatible with eager execution.
at placeholder
raise RuntimeError("tf.placeholder() is not compatible with "
in array_ops.py: line 3341
at NetTrain
xs = tf.compat.v1.placeholder(tf.float32 in /QuantConnect/backtesting/./cache/algorithm/project/main.py: line 57
* add state to backtesting result packets
* self review
* add state in live results
* set runtime error in algorithm
* set hostname in job packets
* Set state when algorithm is null
* address review
* address reviews
* set runtime error when algorithm is not available during runtime
* Fix backtest progress calculation
* Backtest progress monitor class
* Minor changes
* Lazy calculating the processed days count
* Minor changes
* Minor changes
* Feature combo orders
- Add support for combo orders
* Make fill model wait for all grouped orders to emit fills
* Add ComboFill to model multiple fills for combo orders
* Fill combo limit orders
Add some regression algorithms
* Add fill implementation for combo leg limit orders
* Add IFill as common interface for Fill and ComboFill
* Refactor combo orders removing IGroupOrder interface
Move the group order manager to the base Order class
* Update algorithms
* Handle combo order events atomically
* Refactor brokerage transaction event handler
* Refactor combo fill models
* Process fills in batch
* Combo orders fill model tests
* Combo leg limit orders algorithm
* Regression algorithms cleanup
* Fill and combo fill classes cleanup
* Housekeeping
* Refactor equity fill model to derive from base fill model
* Address review changes request
* Handling the new types of orders in the OrderJsonConverter
* Add regression algorithm to test combo orders update/cancel
* Add regression algorithm to test combo orders update/cancel
* Housekeeping
* Address review changes request
* Minor changes
* Security transaction handler method for setting order request id
* Extend public interface for placing combo orders
* Combo order tickets demo algorithm python version
* Tweaks and updates
* Minor fixes
* Minor changes
* Minor fixes
* Address reviews minor fixes
* Minor fixes
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* CryptoFutures adjustments
- Address reviews of CryptoFutures:
- Add new Slice MarginInterestRates collection
- Add new regression tests asserting funding rate application behaves
the same no matter the resolution
- Add auxiliary data by type into the security cache
- Revert BuyingPowerModel changes
* Make interest rate application time deterministic
* Saves Dividend Payments to Security Holdings
Saves information about dividend payments to `SecurityHoldings`. This information will be used to factor in dividend payments to the `Profit` and `NetProfit`.
Adds `SecurityPortfolioManager.TotalNetProfit` to sum all the `SecurityHoldings.NetProfit`.
Adds regression and unit tests.
* Addresses Peer-Review
Improves Regression Test.
* WIP
* Add base currency cash
* Symbol properties and data processing
* Add basic template algorithm
* Add hourly crypto future algorithm
* Minor fixes after live trading testing
* CoinApiDataQueueHandler CryptoFuture support
* Address reviews
* Fix regression algorithms after update
* Allows Market-On-Close Orders Outside Buffer Period
Market-On-Close orders can be submitted before and after the buffer period from 15:45 to 16:00 (Tested with TWS) meaning that we can submit MOC when the market is closed and, consequently, use daily resolution data.
* Adds Regression Test with Extended Market Hours
- Removes `nextMarketClose > Time` condition since it's unnecessary. If the algorithm Time is greater than the close of that day, `nextMarektClose` refers to the next day.
* Updates Unit Test
Updates `OrderQuantityConversionTest` because the MOC orders are submitted. They are placed at 7 pm and invalid before this pull request.
* Updates Summary of new Regression Tests
* Set fill price to zero in OTM exercise orders.
Improved the OTM exercise orders message.
* Update regression algorithms and unit tests
* Add IsInTheMoney property to OrderEvent
* Update SerializedOrderEvent
* Properly setting the option exercise order price to strike price or zero
* Minor changes
* Minor changes
* Fixes Regression Algorithm Testing No Warm Up Requirement
* Removes Volatility Model Warm Up Requirement
If the volatiliy model is not ready, we will use the Brenner and Subrahmanyam (1988) approximation (BS88).
Lean will not set the volatility model if the user doesn't, since the BS88 can handle the null model case.
* Using IV to calculate Greeks, and remove single-step stochastic approximation
* Correct calculation for theta, vega, rho
* Add calculation from Black Calculator and peer review
* Address peer review and added unit test
* Update some tests and correct vega/eho
* Fix Unit Test and Improve Comments
Fixes `IndexOptionCallITMGreeksExpiryRegressionAlgorithm` since `Vega` was really non-zero.
* Fix regression test and add IV calculation
* refactor and bug fixing on peer review
* refactor and bug fixing on peer review
* for rerun test
* add warning on IV estimation not coveraged and edit speed unit test to not exceed 2s per 1000 iteration
* update logging
* update logging and description
* Add default option pricing models and unit tests
* address review
* Added Fed interest rate as risk-free rate with unit tests and set as default for option greeks calculation, added regression algorithms, addressed peer review
* refactor structure of interest rate
* Skip Saturday and Sunday
* regression test fix
* peer review
* Fixes Interest Rate Provider Logic
* Minor tweaks
* Fix start date
* Minor test tweak
* Update interest rates
* Fix unit tests
* Add minor log
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Limit exercise in IB brokerage for limit and cash-settled options
* Add regression algorithm
* Assert that index option is automatically exercised in regression algorithm
* Wolverine LEAN integration
* Wolverine Lean integration - adjusting comments and restrict to Equity only
* fix 0.005$ per share fee
* remove PostOnly from Wolverine as its not needed
* AccountType.Margin and also removed check for USD only currency
* fix Market Wolverine -> USA
* change config
* Minor tweaks
Co-authored-by: Mark Virchenko <mark.virchenko@redduck.io>
* Data monitor (FIRST TEST)
* Calculate data request frequency in data monitor
* Detecting universe data usage in data monitor
* Json data monitor report generation
* Data monitor refactor
* Log and store data monitor report
* Data monitor report tests
* Clean up
* Clean up
* Address changes request
* Address changes request
* Wire data monitor only for backtest, not for live or research
* Minor tweaks
* Some minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Refactor multi IDQH frontier
- Fix for multi IDQH frontier support, specially for IB + QC
- Adding new CompositeTimeProvide. Adding unit tests
* Adjust solution to fix new data available event
* Memory Related Performance improvements
- Make sure we cleanup & dipose of python related objects during pandas
data generation.
- Disable memoizing enumerable use while creating pandas data frames,
since we do not require it
- Reduce unrequired object creations
- Replace concurrentCollections for ordinary locks
* Decimal parsing typo fix
* Add python additional paths for research
* Delegate adding additional paths to pythonpath to PythonInitializer
* Add new way to keep algorithm location at the beginning of pythonpath
* Add switch to disable BeginAllowThreads in PythonInitializer
* Add switch to disable BeginAllowThreads in PythonInitializer
* Address changes request
* Adds Support to Visual Studio Code Dev Container
Adds support and documentation to VSCode Dev Container.
* Address Peer Review
- Adds Dockerfile to install QuantConnect Stubs to enable Python autocomplete.
- Adds settings.json to enable Python autocomplete and LF line ending.
- Removes autobuilder task.
- Fixes Research Laucher
* Launcher and Task Impromevents
Fixes line endings issue
Fixes Launcher to point to right config.json file
Improve readme formatting
* Simplify Jupyter Notebook Deployment
- In some rare cases the FillForwardEnumerator was missing to register
the delisting date of an asset because it was the first data it got.
Adding unit test reproducing issue
* Normalize time in universe behavior
- Normalize Option & Future chain universe behavior regarding their
assets time in universe. They will now respect the universe settings
time in universe value. Adding new regression algorithms asserting the
behavior
* Address reviews & cleanup
- Update QLNet library to latest version from 6/2019 -> 11/2021
- Fix evaluation date which was using settlement date. Adding unit tests
reproducing issue
Use `SetEndDate` to set `EndDate` in `QCAlgorithm` constructor. It will ensure that the `EndDate` is independent of the time the algorithm is executed if `SetEndDate` is not called in `Initilaize`.
Some users don't implement `SetEndDate` to run the algorithm to the latest datapoint, but this is not true if we run the algorithm during the day as the latest datapoint will be 24 hours before the execution time while there is data until the current day midnight.
* Adds Additional Condition to Parse SecurityIdentifier Properties
A string can pass the previous conditions and create an invalid `SecurityIdentifier` (e.g. "Sharpe ration"). If it is invalid, the Market is not supported (e.g. "378").
* Adds Unit Test to Pandas Indexing
* Log Only Once if TryParseProperties Cannot Parse
Use cache to increase speed and remove redundant logging
* docs(VIXDualThrustAlpha): `determine` fixup
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs(VixDualThrustAlpha): `unsubscribe` fixup
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs(BasicTemplateIndia): `Prperties` -> `Properties`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: `Intialization` -> `Initialization`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Get history of a given bar type in python
* Separate GetDataFrame in sub methods to avoid dynamic type resolution
* Housekeeping and unit tests update
* Use all requests in python type method history method
* Address changes request
* Reverted some requested changes
* Handle non-unique multi-index error for Ticks in data frame creation
* Update unit tests
* Update unit tests and add comments
* Add regression algorithm
* Fixes PythonData.EndTime
The `EndTime` property of `PythonData` didn't override the base implementation in `BaseData`. Therefore `EndTime` was always set as `Time`.
Fixes Regression Tests
- The C# version of `Bitcoin` class needs to implement `EndTime`-
- Fixes regression tests
* Minor tweaks and rebase
- Add support for python data types just setting EndTime. Adding unit
tests for PythonData
- Fix for DynamicData EndTime property being fetched. Updating unit
tests
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add algorithm settings to backtest results
* Use algorithm result settings in report generation
* Add unit tests
* Housekeeping
* Changes for live result report
* Housekeeping
* Housekeeping
* Housekeeping
* Add unit tests
* Dynamically set wait time on WS error
- Dynamically adjust wait time on websocket error
* Avoid subscribing multiple times to message events
- Avoid websocket subscription multiple times to message events during
disconnection/connection cycle
* Add Dispose Brokerage WebSocket Subscription Manager
- Add check for live warmup history request, making sure there is at
least room for a single bar between start and end. Avoid sending
start==end when rouded down by the resolution causing some history
providers to fail
- Create SubmitOrderRequest method in algorithm API to reduce
duplication of code and responsability.
- Add order command tests for different failure and success cases
- Fix null reference exception in equity fill model, when there is no
data for an asset.
* docs: `excersizing` -> `exercising`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: fix `exercising for exercising`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: `migth` -> `might`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Support extended market hours in AddFuture()
* Support extended market hours in AddFutureContract()
* Add C# regression algorithm
* Add Python regression algorithm
* Add regression algorithm for future contracts
* Add regression algorithm checking market hour ranges
* Fixed future regression algorithms to use extended market hours
* Fixed future regression algorithms to use extended market hours
* Fixed future regression algorithms to use extended market hours
* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours
* Update future market hours to include extended in market hours database
* Fixed AddFutureOptionContractDataStreamingRegressionAlgorithm to use extended market hours
* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours
* Fixed AddFutureContractWithContinuousRegressionAlgorithm to use extended market hours
* Fixed BasicTemplateContinuousFutureAlgorithm to use extended market hours
* Fixed BasicTemplateFuturesAlgorithm to use extended market hours
* Fix BasicTemplateFuturesDailyAlgorithm to use extended market hours
* Fixed BasicTemplateFuturesFrameworkAlgorithm to use extended market hours
* Fixed BasicTemplateFuturesHistoryAlgorithm to use extended market hours
* Fixed ContinuousBackMonthRawFutureRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureBackMonthRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureHistoryRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm to use extended market hours
* Fixed ContinuousFutureRegressionAlgorithm to use extended market hours
* Fixed DelistedFutureLiquidateRegressionAlgorithm to use extended market hours
* Fixed AutomaticIndicatorWarmupDataTypeRegressionAlgorithm to use extended market hours
* Fixed ConsolidateRegressionAlgorithm to use extended market hours
* Fixed DelistingFutureOptionRegressionAlgorithm to use extended market hours
* Fixed EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm to use extended market hours
* Fixed FutureContractsExtendedMarketHoursRegressionAlgorithm to use extended market hours
* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours
* Fixed FutureMarketOpenConsolidatorRegressionAlgorithm to use extended market hours
* Fixed FutureOptionBuySellCallIntradayRegressionAlgorithm to use extended market hours
* Fixed FutureOptionCallITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionCallITMGreeksExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionCallOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionDailyRegressionAlgorithm to use extended market hours
* Fixed FutureOptionHourlyRegressionAlgorithm to use extended market hours
* Fixed FutureOptionMultipleContractsInDifferentContractMonthsWithSameUnderlyingFutureRegressionAlgorithm to use extended market hours
* Fixed FutureOptionPutITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionPutOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortCallITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortCallOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortPutITMExpiryRegressionAlgorithm to use extended market hours
* Fixed FutureOptionShortPutOTMExpiryRegressionAlgorithm to use extended market hours
* Fixed FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm to use extended market hours
* Fixed FuturesExpiredContractRegression to use extended market hours
* Fixed FutureSharingTickerRegressionAlgorithm to use extended market hours
* Fixed HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm to use extended market hours
* Fixed HistoryWithDifferentDataMappingModeRegressionAlgorithm to use extended market hours
* Fixed HistoryWithDifferentDataNormalizationModeRegressionAlgorithm to use extended market hours
* Fixed LimitOrdersAreFilledAfterHoursForFuturesRegressionAlgorithm to use extended market hours
* Fixed OpenInterestFuturesRegressionAlgorithm to use extended market hours
* Fixed RegisterIndicatorRegressionAlgorithm to use extended market hours
* Fixed SetHoldingsFutureRegressionAlgorithm to use extended market hours
* Fixed WarmupFutureRegressionAlgorithm to use extended market hours
* Fixed AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm to use extended market hours
* Fixed AlgorithmHistoryTests to use extended market hours for futures
* Fixed AlgorithmTradingTests to use extended market hours for futures
* Fixed BrokerageSetupHandlerTests to use extended market hours for futures
* Fixed TimeRulesTests to use extended market hours for futures
* Fixed FutureOptionMarginBuyingPowerModelTests to use extended market hours for futures
* Fixed FutureMarginBuyingPowerModelTests to use extended market hours for futures
* Fixed FileSystemDataFeedTests to use extended market hours for futures
* Fixed QuantBookHistoryTests to use extended market hours for futures
* Split BasicTemplateContinuousFutureAlgorithm to have an extended market version
* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours
* Split BasicTemplateFuturesAlgorithm to have an extended market version
* Split BasicTemplateFuturesAlgorithm to have an extended market version
* Split BasicTemplateFuturesFrameworkAlgorithm to have an extended market version
* Split BasicTemplateFuturesHistoryAlgorithm to have an extended market version
* Revert AddFutureContractWithContinuousRegressionAlgorithm
* Revert AddFutureOptionContractDataStreamingRegressionAlgorithm and added data
* Revert AddFutureOptionContractFromFutureChainRegressionAlgorithm
* Revert AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm
* Revert ConsolidateRegressionAlgorithm
* Revert Algorithm.CSharp/ContinuousBackMonthRawFutureRegressionAlgorithm.cs
* Revert ContinuousFutureBackMonthRegressionAlgorithm
* Revert ContinuousFutureHistoryRegressionAlgorithm
* Revert ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm
* Revert ContinuousFutureRegressionAlgorithm
* Revert Algorithm.CSharp/DelistedFutureLiquidateRegressionAlgorithm.cs
* Revert EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm
* Split FutureMarketOpenAndCloseRegressionAlgorithm to have an extended market version
* Split FutureMarketOpenConsolidatorRegressionAlgorithm to have an extended market version
* Revert FutureOptionBuySellCallIntradayRegressionAlgorithm
* Revert FutureOptionCallITMExpiryRegressionAlgorithm
* Revert FutureOptionDailyRegressionAlgorithm
* Revert FutureOptionPutITMExpiryRegressionAlgorithm
* Revert FutureSharingTickerRegressionAlgorithm
* Revert FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm
* Revert FuturesExpiredContractRegression
* Revert HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm
* Revert HistoryWithDifferentDataMappingModeRegressionAlgorithm
* Revert HistoryWithDifferentDataNormalizationModeRegressionAlgorithm
* Revert OpenInterestFuturesRegressionAlgorithm
* Revert RegisterIndicatorRegressionAlgorithm
* Revert SetHoldingsFutureRegressionAlgorithm
* Revert WarmupFutureRegressionAlgorithm
* Revert AutomaticIndicatorWarmupDataTypeRegressionAlgorithm
* Some cleanup
* Address changes request
* Address changes request
* Add more Class III Milk data to fix DelistingFutureOptionDailyRegressionAlgorithm
* refactor(Algorithm): Correct `Intrinino` -> `Intrinio` references
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(Algorithm): Update `evemts` -> `events`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(BubbleAlgorithm): readability improvements
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(OrderTicketDemoAlgorithm): readability improvements
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Fix KeyError when rendering exposure chart in report with some security types
* Update exposure chart colors and add unit tests
* Updated json test file
* handle multiple command files
* remove default useage
* add tests for verify files are read in order
* add comment
* change reuslt file name
* add comment
* liquidate using given symbol values
* remove redundant
* accept ticker, market, security type to build symbol, to be used by CLI commands
* use null leverage
* symbol can be null if ticker, market, security type values are provided
* reject the default value of security type
* cleanup
* cleanup
* check if status is not invalid
* cleanup
* abstraction to remove duplication
* fix name
* address reviews
* use counter
* add test to validate symbol/ticker agrumnets
* remove hardcoded values
* MRP
* Fix bug
* Rename
* Provide virtual `GetPriceRelative` function to override for any variants in future
* SymbolData class within model
* Address peer review
* fix bug
* revise unit test and fix bug in python model
* revise unit test and fix bug in model
* Revise regression statistics
* Revise regression statistics
* Revise regression statistics
* Revise regression statistics
* Revise regression statistics
* Address peer review
* Updated unit tests according to peer review
* Address peer review
* Fix LocalMarketHours.GetMarketClose to get the actual next market close time with extended market hours
The method was not properly considering gaps between regular hours close and extended market open
* Revert some changes to fix errors
* Add performance gh action
- Add new performance github action, running python and csharp benchmark
algorithms
- Adjusting algorithms length so they are not too long or too short
* Checkout master branch benchmark reference
- Fill model will take into account internal configurations, this is
specially relevant for continuous future mapped contracts. Updating
regression algorithms and unit tests.
- Allowing internal configurations to update trade builder information
* Remove suppport for market orders and MOO orders for futures
* Add regression algorithm to assert that limit orders are filled on after hours for futures
* Remove suppport for market orders and MOO orders for futures in DefaultBrokerageModel
* Update regression algorithms stats
* Fixed regression algorithm and clean up
* Add unit tests
* Fix regression algorithms
* Address changes request
* Respect warmup resolution given
- The data feed will respect the warmup resolution given and override
the resolution used by the algorithm when adding a subscription.
Updating regression algorithm to keep previous statistics. Adding new
regression algorithm asserting the desired behavior
* Testing improvements
- Add more unit tests and regresion test
- Add missing data for crypto
- Fix bug with FFed data crossing after the end time of the warmup
request
* Add more Warmup resolution regression algorithms
- Adding more warmup resolution regression algorithms, using
Settings.WarmupResolution and an option selection case
* Add more warmup regression tests
- Adding more warmup regression tests.
- Will no longer skip universe selection subscriptions from warmup
resolution enforcement. Updating regression algorithms data points
* Fix bug with data rounding
- Fix data rounding bug when warmup resolution is set to a different
value than the original configuration. Updating regression algorithms
to assert the expected behavior
* Address reviews
- Revert regression algorithms changes to use Resolution during warmup.
Updating their stats.
- Adding new regression algorithms asserting the behavior warming up
using a timespan and no warmup resolution
- Fix bug where data used to warmup the 'normal' enumerator will make it
through into the warmup time span. Updating tests
* Address reviews
- Add missing comments, explaning warmup algorithms time span
calculations.
- Revert changes in existing `WarmupOptionTimeSpanRegressionAlgorithm`
to reduce diff to minimum
- Adding new warmup unit tests asseting algorithm warmup start time, for
different combinations of bar count, timespan, resolution
* First commit
* Any change
* Add more unit tests
* Nit changes
* Changes requested
* Fix bugs and requested changes
* Requested changes
* More changes
* Check ongoing backtest orders
* Read ongoing backtest does not fail
* nit change
* Fixes for standarized API orders response
- Fixes for a standarized API read orders reponse
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Minor improvements
- Avoid division by zero error in backtesting result handler. Race
condition where the SetAlgorithm implementation would still be running
and we try to loop through the update method
- Improvement nullreference exception message
- Cleanup resolution to timespan extension method
* Address reviews
- Cleanup data config fill forward conditional statement for tick
resolution
* Fix live future and option universe selection daily resolution
- Adding unit test reproducing issue for option and future universe
chain using daily resolution, since was rounding down in UTC TZ the
date might not change and it would re emit the same data point
* Address reviews
- Add more test cases
- Adding locks around RegisteredSecurityDataTypesProvider to allow it to
be used by multiple threads at the same time without exploding
* Minor fixes for mapped futures contract
- Future security was never able to be non tradable
- Skip remove future contracts from live holdings. Adding unit test
reproducing issue and regression algorithm
* Fix removal of continuous futures
- The canonical continuous future would leave behind an OpenInterest
subscription when removed because of a different in the way the
configurations were built, this will now be centralized in a single
method. Adding unit tests reproducing issue
- FillForwarding setting of the continuous future was not being
respected. Updating algorithm reproducing the issue to assert the
behavior failing in master
* Address reviews
- Minor refactors addressing reviews
* Fix result handler live holdings
- Live result handler live holdings will include future assets
associated with continuous futures. Adding unit tests
* Fixed for live holdings price rounding
- Live holdings will use the SPDB minimum price variation for rounding.
Adding unit tests
- Remove thread in charge of sorting and updating the queues weight.
This will be handled by each worker thread instead. Avoid cases where
the work count is so high that the sorting thread in master would not
be able to update all the weights, causing data to be enqueued beyond
the max prefetch limit. Adding benchmark algorithm reproducing issue
- Smaller live warmup history requests, will keep track of the last point
we got from the file based enumerator and start our history enumeration from this point
* Add new Python QCAlgorithm.History() method with all parameters and type
* Add regression algorithms
* Using all parameters in History()
* Use private methods to reuse History() code
* Use private methods to reuse History() code
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Add unit tests for QCAlgorithm.Python.History()
* Asserting history count
- Similar to the OptionChainUniverse, the future chain universe should
also add it's contracts using the raw data normalization mode. Adding
regression algorithm reproducing issue
- Replace logic to convert TimeSpan based PeriodConsolidator into count
based, by an override of the 'potentialStartTime' in the case we could
be falling into a look ahead consolidated bar end time
- Adding OpenInterestAnnual where the contract maps when any of the back month
contracts of the next year have a higher volume that the current front month.
- Add support for backwards compatible data mapping mode additions
- Updating ES map and factor files. Adding a little daily data too
- During warmup period the algorithms initial time might not be a rounded date
value, so it's important to take into account hours/minutes. Adding
regression algorithm reproducing and asserting issue
* Period timespan consolidation improvements
- If user is trying to consolidate a period providing data of a bigger
period we will now throw an exception. Adding tests
- If both consolidated and given data share the same period, gently
adjust the consolidator into a data count of 1. Adding tests
- Fixing bug in QuoteBarConsolidator period double accounting. Adding unit tests
* Add Period and Count regression algorithm
* Fix QuoteBar NonTimeBased Consolidator Period
- Fix QuoteBar non timebased consolidator period, that was accounting
twice for the initial bar period. Updating unit tests
* Improve QuoteBarConsolidator assertion
- Assert quote bar consolidated time and endtime in unit tests
- Add error log at OptionSymbol.IsOptionContractExpired in the case we
detect an unexpected expiration time. That could be related to
Symbol.ID.Date being incorrect
* Fix option price and greeks always beign zero on the expiration date
* Fix option price and greeks always beign zero on the expiration date
* Updated documentation
* Style and performace changes
* Address changes request
* Fix QL.Settings.includeReferenceDateEvents not being set on every thread
* Updated code documentation
* Add indicator for allowed option styles to QL option price models
* Add and update option price model tests
* Update option price models methods sumary to indicate allowed option styles
* Add regression algorithms for option price models for different option styles
* Update OptionPriceModel regression algorithms to use Lean local data
* Add Python regression algorithms for option price models for different option styles
* Update OptionPriceModel regression algorithms to assert that greeks are valid
* Address changes request
* Address changes request
* Update OptionPriceModel regression algorithms to check both call and put contracts
* Update OptionPriceModel regression algorithms to use correct test data
* Update OptionPriceModel regression algorithms to throw in OnData
* Address changes request
* Update OptionPriceModel regression algorithms to assert greeks are not all zero
* Live T-1 selection warmup
- Only use BaseDataCollection class
- Reuse collection enumerator
- Remove FuturesChainUniverseSubscriptionEnumeratorFactory
- Universe selection will use Cache providers
- Add null reference check
- Add more tests
- Fixes for warmup fill forwarding. Add more tests
* Address reviews. Add missing comments
* Refactor chain providers
- ChainProviders will use quote, open interest and trade information,
which ever is first. Updating regression algorithm changing contract
being added.
- Add base BacktestingChainProvider for shared logic, reducing
duplication.
- ChainProviders will now use the given IDataCacheProvider and
IMapFileProvider
- If Providers are called for a date the exchange is closed they will
search for the previous available date where the exchange is open.
Adding unit tests.
* Address reviews. Logging improvements
* use research-object-store-name key from config while creating object store for research notebooks
* initial removal
* remove reference
* move test to private repo containing downloaders
Since there is `UpdateStopPrice`, `UpdateLimitPrice`, `UpdateQuantity` and `UpdateTag` method to make it easier to update the `OrderTicket`, we include `UpdateTriggerPrice` to cover the `LimitIfTouched` case.
Also, updated the docs or the arguments for these methods.
Updates LimitIfTouchedRegressionAlgorithm:
- Adds `UpdateTriggerPrice` call that rounds down the `TriggerPrice`.
* use research-object-store-name key from config while creating object store for research notebooks
* remove bitfinex files
* remove redundant ref
* move test to private repo
* Warmup minor fixes
- Result handler fix
- Centralize and normalize status update during warmup
* Address reviews
* Minor adjustments
* Add console message during warmup period
* Fix warmup percentage update
* Overload PandasConverter.GetIndicatorDataFrame to accept a Python dict
* Shared implementation code for PandasConverter.GetIndicatorDataFrame overloads
* Added documentation for private shared methods used by PandasConverter.GetIndicatorDataFrame
* Added unit tests for PandasConverter.GetIndicatorDataFrame
* Added unit tests for PandasConverter.GetIndicatorDataFrame Dictionary overload
* Address change requests
* Address change requests
* Reproduce TrailingStopRiskManagementModel bug with test case
* Fix TrailingStopRiskManagementModel to make it relative to max price
* Adapted old TrailingStopRiskManagementModel test to new implementation
* Fix TrailingStopRiskManagementModel Python version
* Fixed TrailingStopRiskFrameworkAlgorithm regression tests data
* Handling both long and short positions in TrailingStopRiskManagementModel
* Traking holdings value instead of unrealized profit in TrailingStopRiskManagementModel
* Checking for position side change in TrailingStopRiskManagementModel
* Handling immediate liquidation in TrailingStopRiskManagementModel
* First commit
* Add unit test
* Change implementation
* Add unit tests
* Nit change
* nit change
* Remove unnecessary methods
* Add more unit tests
* Revert "Add more unit tests"
This reverts commit 1ba2ab7454.
* Add more unit tests
* Add more unit tests
* Improve Implementation
* Change unit tests
* Remove unit tests
Remove unit tests from AlgorithmRegisterIndicatorTests.cs
* Coarse fine flag improvement
- Coarse will check global data folder for fine existance if it's
different than the given path to check
* Update readme.md
- Minor fix for FuncPeriodSpecification so it always uses the same
DateTime to assert the given function. Improve documentation and
exception message being thrown
* Add missing stablecoins
* Fix bugs
* Remove MIMUSD stablecoin pair
That pair is already in the SPDB
* Add more stablecoins pairs
- Add SUSD and IDRT
* Fix bug
* Requested changes
The algorithm posted in the GH was failing because when finding a conversion between currencies BTC to USD, it wasn't taking into account BTCUSDT and that USD = USDT because is a stablecoin
* Solve bugs
* Nit changes and more tests
* Requested changes
* Nit changes
* Requested changes
* Nit change
* Allow DataWriter to merge existing ticks if desired
- Optionally allow lean data writer to merge new ticks with existing
- Adjust ZipStreamWriter
- Fix bug in compression lib that was leaving files open. Reproduced by
added tests.
* Address reviews
- Define new WritePolicy to configure the behavior of the LeanDataWriter
regarding how it should handle writing to a file: merge, overwrite,
append
* make public safe to expose test properties
* extract ib files
* remove dependencies
* refactor to use ib reference as string
* remove un-used dependencies
* remove redundant refrences
* Added checking if algorithm is stopped in initialize method
* Quit on initialize adjusment
- Minor adjustments to solution. Adding more regression algorithms
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Update Tick.cs Time parsing
* Adjust equity tick millisecond read
* Fix tick sub millisecond precision
- DateTime.AddMilliseconds will rount to the nearest integer. So instead
use AddTicks
* Avoid sporadic test failure
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* set price currency for open orders from brokerage setup handler, remove setters from brokerage
* order already has price currency being set by brokerage transcation handler
* remove as overwritten by BrokerageSetupHandler.GetOpenOrders
* refactor because can't use internal setters in private repos
* refactor, use ctor to set attributes
* remove redundant broker dependency
(cherry picked from commit 57b071eb174ca9fe857021ebffa7a546834e86f7)
* check limit price for limit order instead for checking price
* add price agrument for MarketOrder
* remove, not required anymore
* AddUnrequestedSecurity should not return null
* assert string in not null cases
- Fully reset the security cache when it's finally removed from the algorithm.
Adding regression algorithm reproducing issue
- Updating regression algorithms which would trade based on data
previously available
* Warm up EmaCrossAlphaModel
Warm Up EmaCrossAlphaModel indicators
* Fix regression test bug
When using the default `EmaCrossAlphaModel()` the period of both indicators to be ready is bigger than the difference between the start date and the end date of the algorithm. Then, as the algorithm didn't warm up the data both indicators of EmaCrossAlpha never were ready, but now as the model warms up the data both indicators are ready so we get different statistics
* Requested change
* Fix unit tests
As there wasn't items in `AddedSecurities`, when trying to remove the items in ´RemovedSecurities´ there was nothing to remove because there was never a security in `_symbolDataBySymbol`. That's why, in order to test, the behavior of `EmaCrossAlphaModel` when removing a security we need to first add one to then remove it.
* Requested changes in Python
- Requested changes in Python
- Nit changes
* Nit change
* Requested Changes
* Add RemoveConsolidators() method in Python version
* Add requested History overload
* Add more regression tests
* Revert "Add more regression tests"
This reverts commit 71b279e917.
* Add more regression tests
* add data count properties
* 'add history count property
* assert data counts
* update missing override
* consider override/virtual cases
* implement data count
* add message handler for regression tests
* use regression test message handler
* set algorithm manager for regression test message handler
* update data count
* check if stats are present, check if algo manager is not null
* update
* add c# algo
* make same as c# algo
* use new line
* logic shifted to RegressionTestMessageHandler
* cleanup
* auto cleanup
* skip non deterministic data count
* change data count
* use inheritance
* improve stats
* update couht
* add sma indicator to c# and customSMA to python
* call base method before executing further
* skip test
* revert to original
* add duplicate sma
* skip regression test
* Make continuous futures untradable
- To match live trading behavior. Adjust continuous futures securities
to be untradable
* Set ContinuousFuture as non tradable
- Set continuous futures as non tradable. Update regression algorithms
- Fixes for symbol capacity calculation
`WarmUpIndicator` for Python indicators doesn't return the indicator anymore after #6027. So all overloads should return `void`.
Fixes `SmaCrossUniverseSelectionAlgorithm` [C# and Py].
* Added NYSELIFFE exchange support to Interactive Brokers brokerage.
* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].
* Added NYSELIFFE exchange support to Interactive Brokers brokerage.
* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].
* Added NYSELIFFE exchange support to Interactive Brokers brokerage.
* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].
* - Added NYSELIFFE futures to Futures.cs
- Added method GetGoodFriday() to calculate Good Friday to FuturesExpiryUtilityFunctions.cs.
- Added unit tests for GetGoodFriday().
- Added NYSELIFFE futures expiries to FuturesExpiryFunctions.cs which uses GetGoodFriday() instead of hardcoding the date of Good Friday in the market-hours-database.json.
- Added unit tests for NYSELIFFE futures expiry.
- Added NYSELIFFE to Exchange.cs
- Removed Equity-nyseliffe-[*] and Index-nyseliffe-[*] from market-hours-database.json
- Updated Option-nyseliffe-[*] to FutureOption-nyseliffe-[*] on market-hours-database.json
- Added the supported NYSELIFFE futureoptions to symbol-properties-database.csv.
* Removed blanks after India tuple
* Added the dates for 2022 and 2023 to the market-hours-database.json.
Removed "FutureOption-nyseliffe-[]" from market-hours-database.json.
Added to FuturesOptionsSymbolMappings.cs:
{ "YG", "OYG" },
{ "ZG", "OZG" },
{ "ZI", "OZI" }
* Minor tweaks for nyseliffe FOPs
Co-authored-by: Chris Coetzee <chris@polidata.ai>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Deprecate breakpoints
Drop Watchlist
Remove breakpoints from lean
Prep install of NetCoreDbg
Cleanup
* Install NetCoreDbg
* Update Research Images with Newer Jupyter
* Binance.US base changes
* biniance exchange info update
* Binance US uses same fee rates as main Binance
* binance us brokerage model tests
* use base implementation of GetBuyingPowerModel method
* add gdax exchange info downloader
* add downloader method to program]
* fetch currency description
* change definition to include headers
* use extension method to make request
* remove log from test
* replace WebRequest
* cleanup
* use relevant name
* implement IExchangeInfoDownloader for bitfinex, initial commit
* add default values
* use default market value
* use correct attribute for lotsize
* don't skip missing values
* handle multiple downloaders
* add gdax and bitfinex exchange downloader
* follow LEAN data directory structure
* update SPDB
* order tickers
* order tickers
* add exchange info downloader test template
* delete files
* update SPDB
* use currency mapping
* update bitfinex symbols
* update currency mapping
* sort result after old currency symbols are used
* use market of the respective brokerage
* no more unknown symbol
* change minimum order size value
* direct conversion possible
* update bitfinex symbols
* change user-agent
* add test for indirect conversion
* update stats
* remove un-used
* use utc time for slice sync
* use utc time
* refactor
* add regression test
* use utc time
* use utc time
* make utctime required parameter
* add utcTime in slice creation
* assert warm up complete
* check if algorithm is still warmingup
* use exchange tz
`CashBuyingPowerModel`, which reflected on `AlphaStreamBrokerageModel`, a margin-only brokerage.
It also didn't consider the account type, so `InteractiveBrokersBrokerageModel` was using the margin model even if `AccountType.Cash` was selected.
Removes `GetBuyingPowerModel` method from other Brokerage Models when their cases are covered by `DefaultBrokerageModel`
Fixes some typoes in `TradierBrokerageModel`
Fixes unit and regression tests. For the regression tests, we have explicitly set the brokerage model.
- The BrokerTransactionHandler will also adjust virtual positions for
margin accounts when fees are in base currency and the asset is a
crypto or forex pair. Adding new regression tests reproducing issue
* extract binance brokerage
* remove binance tools from main toolbox app
* modify OrderTestParameters constructor
* fix BinanceBrokerage ref config.json
* update symbol properties
* move BinanceFeeModel tests to the right place
* initial commit
* get history from data providers
* merge history from history providers
* merge slice
* Add merge function in slice
* Update test
* merge aux data
* add data points in _data
* append new data points in original list
* Add test suite for HistoryProviderManager
* reduce complexity
* setup once
* add fake history provider
* doesn't count aux data
* add tests for options
* style changes
* add custom data
* merge rawDataList
* use array of history providers
* optimize
* fix formatting error
* add comment
* add tests
* simplify
* use list
* use abstraction
* split tests
* add test
* use abstraction to create generic enumertor class
* refactor
* accept list of type T
* sync history slices
* use SubscriptionDataReaderHistoryProvider for live
* rename test file
* return empty
* add tests
* cleanup
* address reviews
* optimize
* Fix method definition
* use initial time for basedata
* refactor
* re-use collection
* inherit HistoryProviderBase
* always return HistoryProviderManager
* add tests
* update rawDataList
* reset composer
* consider null elements
* add tests for binary search method
* Follow lean coding style
* revert
* remove binary search method
* convert to field
* Update MHDB and SPDB
Update MHDB and SPDB with micro futures
* Fix bugs
* Add more FutureExpiryFunctions
* Requested changes
* Nit changes
* Nit changes
- Fix some future values in SPDB
- Test LastFriday() method
- Apparently Micro CHF/USD Futures(MSF) refers to the micro of Swiss Franc Futures (6S). The same happens with Micro JPY/USD (MJY) Futures and Japanese Yen Futures (6J)
* Fix bugs
* Micro futures MultipleFactor
* Fix some futures MultipleFactor
* Adding unit tests reproducing issue.
* Fix a couple of minor bugs
- IsMarketOpen will work correctly when used with daily and hourly
resolution.
- slice.Get will work correctly with python custom data
- ExtendedDictionary will be able to dinamically access methods,
required for python and private C# data types
* Refactor solution. Add more tests
* Remove unrequired import statement
* WIP
* Fix fees in base currency not being subtracted
- Fix fees in base currency not being subtracted from the quote currency
for crypto cash accounts. Updating regression tests to assert
portfolio, cashbook state and holdings state.
* Fix unit test race condition
* Use GC server mode for tests
* Fix daily auxiliary data points emission time
- Due to fillforwarding, in some cases with daily resolution symbol
change events (generically any auxiliary data) would arrive late.
Updating regression test to reproduce the issue. Adding unit test
- Some refactoring and logging improvements
* Address reviews
* Remove old xml docs param
* change BinanceBrokerageModel
* supply spot vs margin endpoint as parameter
* wip
* send margin order for margin account
* "NEW" means that the order has been accepted by the Binance engine.
* fix cash balance
* use JsonConverters for account parsing
* unit tests
* fixup
* lazy connect
* fix connection
* allow api client to be null if DQH only
* make method private
* more unit tests
* fix Dispose
* fix tests
* fix IsConnected condition
* run test as additional
* add some comments
* improve unit tests
* improve null checks
* tidy up the code
* Refactor `GetFilePath()`
Add also useful methods to use with this one
* Nit changes
* Requested changes
* Requested changes
* Restore SaveString()
* Nit changes
* Address self review
* Test improvements
* Adjust example KerasNeuralNetworkAlgorithm
* Minor tweak for KerasNeuralNetworkAlgorithm.py
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Binance fees deducted from fill quantity accordengly
- For Binance cash accounts while buying, if fees are from the base
currency of leans virtual position, we need to deduct the fee from the
fill quantity, else we can end with a position bigger that it actually
is and not be able to liquidate
* Refactor solution
- Refactor solution into a more generic approach solving fees in base
currency at the BrokerageTransactionHandler level, covering all
brokerages that require it. Adding regression algorithm reproducing
issue.
- Update Bitfinex and Binance fee models to correctly reflact reality
* Log fill quantity adjusment once
* Add support for live price scaling
- Add support for live trading price scaling for continuous futures.
Adding unit tests
* Move live price scale application. Updating unit tests
- Live trading application of price scaling will happen before fill
forwarding and updating securities real time price. Updating unit
tests to reproduce issue
- GetLastKnownPrices will no longer guess which resolution to use but
rely on other methods implementation/
- Updating basic template future algorithms to warmup contracts and
assert it
- Minor improvements for FunSecurityInitializer and FuncSecuritySeeder
* Always reuse aggregator instance if any
- When fetching a IDataAggregator instance from the composer, do not
enfore type name on existing instances
* Fix unit tests
* add data
(cherry picked from commit 814011d89e5316d150f88ffca5f48d8d5f0ea7d9)
* update market hours for index
(cherry picked from commit edac40732c120eb84d27de00594b59eebb4983f5)
* add index algorithms
(cherry picked from commit b22d27b4fa98172c435f7c26de4a3a297c49a6b7)
* update statistics
* add cash
* Add india market
* add leverage for index
* can subscribe to index
* update format
* fix wrong cash
* fix ticker names
* update data
* update ticker and stats
* update docs
* specify enpoint url of ftx api (can be FTX pr FTX.US)
* more metadata for ftxus
* tidy up code
* tests
* more tests
* fix tests
* fix us fee rates
* add account tier
* update symbol props
* typo
* typo-2
* update symbol properties
* use FTXUS fee model
* minor tweaks
* replace to local functions as they are more performant
* fix random generator upper bound
Next() includes minValue, but not maxValue, so we increment it +1
* introduce abstract layers
* refactoring
* fix tets
* adapt tests
* fixup
* implement blackschole price model for options
* use risk free rate
* use ql price model
* wip
* change interface
* fix
* tidy up the code
* wip
* iterate groupped symbols
* wip
* wip
* fix
* allow symbol of different types
* improve settings
* wip
* iterate full range
* fix issue with negative option
* fix
* fixup
* use StandardDeviationOfReturnsVolatilityModel
* re-use existing tick types per security type
* parametrize underlying security type
* use default option style
* dynamic option price model
* fix enumeration
* test
* fix unit tests
* refactor code
* remove unused file
* minor tweaks and refactoring
* rename symbol generator class
* fix interface
* add comments
* more comments and unit tests
* more tests
* add disclaimer
* more tests
* more comments and tests
* split tests into different files
* tidy up the code
* tidy up the code; more tests
* refactor TickGenerator => use security price directly on each iteration
* remove dupe; reuse main constructor
* use SecurityManager, refactor code
* bugfix: save ticks in history array
* check volatility warm up & tests
* more unit tests
* describe volatility period span in settings
* rename command line option
* Minor adjusments. Address review
- Use Lean log handler instead of writting directly to console
- Rename BlackShcolesPriceGenerator to generically OptionPriceModelPriceGenerator
- Minor format clean up & standarization
- Add support for specifying the option chain size
* Rename TickGenerator private fields
* Fix unit tests
* fix tests class name
* Support tickers being specified
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Zip entries will be sourced from cache provider
- Option underlying will use SubscriptionDataSource to fetch it's data.
Fixing bug where it would let through an old data point, or miss
sending data through.
* Add india market data
* use local data for algo
* Add India market regression algo
* update india market data
* Update readme
* add python algo for BasicTemplateIndiaAlgorithm
* Add India data regression python algo
* data india data files
* update tickers
* Fix algo template
* remove data
* fix stats
* update stats
* remove unused data
- Live subscription enumerator will clone the underlying data set when
live mapping is being done. To avoid issues where IDQH implementations
could reuse a data point with same configurations. Adjusting unit test
to reproduce issue
* Fix typo
* Fix TryGetValue to return false if key is not found
* Revert "Fix TryGetValue to return false if key is not found"
This reverts commit b85b7b579a.
* Update documentation for TryGetValue
* Move Optimizer-related DTOs and JSON converters into Common/Optimizer
* Add REST methods for Optimization
* Move OptimizationStatus into Common
* Change optimizationId parameter type to string
* Update Optimization and add lightweight optimization object
* Rename lightweight optimization to BaseOptimization and remove unneccessary properties
* Remove snapshotId from Optimization, add ParameterSet to Backtest
* Add missing IApi.cs method signatures
* Move ParameterSet into Common
* Replace Backtest with OptimizationBacktest
* Update UpdateOptimization to not include null or empty name and layout params in the request
* Change Objective targetTemplate regex pattern from ['(.+)'] to (.+) to prevent escaping target strings without whitespace
* Return Estimate object when calling EstimateOptimization
* Use DefaultNamingStrategy when serializing constraint operators
* Revert "Change Objective targetTemplate regex pattern from ['(.+)'] to (.+) to prevent escaping target strings without whitespace"
This reverts commit fbe7de0fd7.
* Update Api method signatures
* Add unit tests
* Fix XML comment referring to the old class name
* Fix XML summary for OptimizationResponseWrapper
* Address review feedback
- Remove unused testOrganizationId
- Change NodeType from string to NodeType enum
- Clarify unit types for Estimate time and balance
- Simplify JsonConverter classes
* Add accessors to Common/Api classes
* Define performance metrics names in PerformanceMetrics class
* Remove unnecessary branching logic from GetSeriesValues method
* Add crefs and examples to XML comments in the Api class
* Revert "Change NodeType from string to NodeType enum"
* Remove layout param from UpdateOptimization method
* Backtest property ParameterSet should be of type ParameterSet
* Add asserts for deserialization in OptimizationBacktestJsonConverterTests
* Replace the three target-related properties with Criterion
* Add serialization and deserialization tests for Optimization
* Remove Optimization Serialization test
* Add EstimateDeserialization test
* Add asserts for integration tests
* Address self review
* Revert test case
* Update Nodes.cs
* Update Nodes.cs
* Set Aborted status when Optimization fails to start
* Add ParameterSetJsonConverter and ParameterSetJsonConverterTests
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Do not send internal SecurityChanges to Algorithm
- Will not send internal security changes to the Algorithm by default.
Following custom security changes filter pattern. Updating regression
algorithms to assert behavior.
- The universe member will know wether it was added with internal
configurations or not
* Address reviews use a separate collection for internals
* Refactor solution. Adding security changes constructor class
* GH-6115: adjust dates
* fix
* use another approach
* Add lean data writter multiple days data unit test
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Handle security added and removed in the same loop
- Correctly handle adding and removing a security in the same loop.
Adding regression test
* Adding comments
* Align universe security and configuration TZ
- Make sure universe security and configuration tz are aligned always.
Adding unit test reproducing issue
* Address review UserDefinedUniverse will use UTC TZ
* CompositeDataQueueHandler handles duplicate configs
- CompositeDataQueueHandler will handle duplicate subscriptions sharing
the same config. Adding unit tests
* Refactor bug solution after review
- LiveSubscriptionEnumerator will not long perform any symbol mapping
but will just trigger a new subscription call when remapped.
- CompositeDHQ will handling symbol mapping and keep track of these
mapped configs to trigger unsubscribe accordengly. Adding unit tests
* Adjust solution after review
- To avoid breaking uniqueness of SubscriptionDataConfig will be adding
a private `mapped` property that will change equality of mapped
configs to avoid them clashing. Adding unit tests
* Rename IDQH subscription extension methods
* Minor performance improvements
* CompositeDataQueueHandler is UniverseProvider
- CompositeDataQueueHandler implements the UniverseProvider interface.
Adding unit tests
* Indicator
* Fixed logic error
* Removed old files, added KER into KAMA
* Removed old comments
* Added requested changes
* Minor clean up
* Refactors KaufmanEfficiencyRatio and KaufmanAdaptiveMovingAverage
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
* Updates KeltnerChannels to Update MiddleBand with EndTime
* Adds Unit Test
This unit test shows that the timestamp of the MiddleBand and the indication are aligned as expected.
* initial commit
* Follow IDQH implementation
* Expect a list of data handlers from LiveNodePacket
* Return null if can not subscribe
* refctor to add check for subscription
* initialze null
* Add tests
* Check subscribe retuns null/not-null
* cleanup
* Read all required IDQH credentails to job
* Use CDQH to handle all IDQH instances
* constructor abstraction to call from setjob
* use flag
* remove redundant because derived will call initialize on it
* abstract and initialize from setjob
* handle null enumerators
* get creds from data handlers
* handle single data handler value from data-queue-handler
* Fix to support a json array
* Fix missed constructor call
* change access modifier to access from Tests files
* Add test to get brokerageFactory from dataQueueHandler
* Fix init flag to handle all conditions
* Add docs
* initialize from setjob
* Check if websocket open before using
* change defination of initialzie to include tradier
* clean up
* change defination
* fix wrong api key name
* return empty enumerator
* check websocket open before sending request
* check connection before subscribing
* fix to include more cases
* check websocket open before sending request
* Minor refactoring
* reafctor and use IsConnected
* remove unused
* clean up
* Fix test cases
* reverse change
* include config changes
* connect to websocket from setjob
* check websocket connection from setjob
* clean up
* include condition for IDQH that are not brokerage
* Address review
* Add market check condition before subscribe
* Remove deprecated
* Minor fix for deserializing data queue handler
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Fix for adding continuous future and manually adding contract at the
same time. Continuous future was using the user defined universe
symbol and caused clashes. Adding regression test reproducing the issue.
- Fix for IB option position update race condition, where the BTH would
incorrectly assume it should trigger an early option assignment. Adding unit tests
* Give `code()` more time per iteration
* Another proposed solution
* Revert "Another proposed solution"
This reverts commit 29bf5a7554.
* Add an AutoResetEvent field in `TimeConsumer()`
* Avoid change TimeMonitor implementation
* Remove changes in `TimeConsumer.cs`
* Requested changes
* Avoid re-implementation
* Continuous Futures Refactor. Live Mappings
- Adding support for live mappings. LiveTradingDataFeed will handle any
symbol mapping at the configuration layer and resubscribe through the
IDQH
- Refactoring continuous futures adding ContinuousFutureUniverse that
will select the currently mapped security
* Minor fixes
- Remove addition of configurations in UniverseSelection step, leave
resposability for universe.
- LiveTradingDF future unit test will only assert slice data for non
internal feeds.
- ContinuousContractUniverse will respect internal option interest
subscription
* Address review
* Properly handle pending CancelOrderRequests
To avoid unneccessary "Insufficient buying power to complete order" errors showing up as Invalid orders we need to skip further processing of pending CancelOrderRequests in the BacktestingBrokerage and instead let them be properly removed during the next transaction handler run.
* Update BacktestingBrokerage.cs
* Activate Index and IndexOptions low res
* Add Index and IndexOption low res data
* Add IndexOption regressions
* Fix bug with IndexOption QuoteBar data not being scaled
* nit - update comment for _scaleFactor
* Update data and regressions to reflect fix in data
* Tweak regressions
* Fixes for FutureOptions support in LeanData
* Add CreateCanonicalOption() utility function for Symbol.cs
* Add aggregated Futures/FuturesOptions data to Lean
* Add FutureOptions regressions for daily/hourly data
* Allow Futures to be added with low resolution
* Add Future regressions using hour/daily data
* Nit - Python Class names
* Add reviews
* Add alias into CreateCanonicalOption
* Add tick type to data downloader get (price data) call in order for api's with rate limits to handle not making an api call at all.
* Added DTO object to pass into data downloader get
* Added missing header for new file
* Address review. Minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix for python enumerable data history request
- Fix for python enumerable type data history request. Adding unit test.
* Add methods for adding data points into a baseDataCollection
* Move processing of delistings to Brokerage
* Deal with case that exchange is not open on OptionSymbol.ID.Date
* Refactor solution to use DelistingNotification event
* Adjust some regression expected liquidation time
* Mark some todos on deprecated functions
* Update expected liqudation time for Py regressions
* Update regressions that have been validated
* Use HandlePositionAssigned for assignment orders
* Update regressions
* Update some missed unit tests; remove one that is already covered by regression
* Cleanup deprecated backend functions
* nit - small cleanup adjustment
* Post rebase fix
* Address review
* Minor tweak to py regression
* Super Trend Indicator #4653
* Updated test data
Previous test data was wrong
* Reduced if statement
* Updated tests
Replacing spy test data with dwac test data from trading view.
* Minor comment update
* Minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* First BetaIndicator prototype
- In construction
* Fist BetaIndicator version and unit tests
* More unit tests and regression test
* Nit change
* Requested changes
* Nit changes
* Requested changes
* Adjust beta formula slightly and nit changes
* Nit change
- Add new CustomWeight PCM for alpha streams
- Add new AlphaStreams AlphaModule that will handle security additions
and removals, removing this logic from AlphaStreamsBasicTemplateAlgo
* Remove user plan enum
- Get maximum order and runtime from job controls
- Remove user plan enum definitions
* Add log for BacktestingSetupHandler maximums
* Create generic writing for LeanDataWriter, + notes on todos
* Make Options Daily/Hourly data store by year
* Refactor Generic Write
* Permit hour and daily resolutions for options
* Refactor writer to merge when needed with other files
* Cleanup redundancies, run write tasks in parallel
* Make needed classes/vars available
* Update tests to reflect new naming convention for daily hourly options data
* Add Byte[] overloads for ZipData functions in compression
* Implemented Store() for ZipDataCacheProvider
* Have LeanDataWriter use a DataCacheProvider
* ZipDataCacheProvider cleanup
* ZipDataCacheProvider tweaks, doesn't support storing non-zips
* Test adjustments
* Update LeanDataWriter to use Write instead of SaveDailyHourly/SaveMinuteSecond
* Implement tests to verify DownloadAndSave behavior
* Nit cleanup on DownloadAndSave tests
* Fix for options daily/hourly underlying equity subscription read
* Add daily/hourly options data and regressions
* Add missing open interest for hourly
* Fix writing of OpenInterest Daily/Hourly data
* Update data
* Fix Date typo in regression
* Use daily algorithm to test delisting
* Revisions part 1
* Expand test for DataCacheProviders; refactor DiskDataCacheProvider
* nit - test adjustments
* ZipDataCacheProvider test setup refactor
* Adjust multithreaded read/write test; fixes for ZipDataCacheProvider
* Move DiskDataCacheProvider to its own file and add write test
* Remove _appendToZips; always overwrite entry or create zip
* Add mapping regression for daily options
* nit - add license to regression
* Fix Tick write case where more than one data point for a DateTime
* Fix data issue
* Address review
* Tweaks for tests
* Stop Store() early if no entry name is given
* Warm up MACD indicators
- When a security is added in the MACD alpha model, it's warm up at once
* Add unit tests
* Nit change
* Code style and nit changes
* Implement IIndicatorWarmUpPeriodProvider
- Implement IIndicatorWarmUpPeriodProvider in PythonIndicator.cs
- Make a unit test to check whether the WarmUpPeriod is working as expected
- Make a regression test to check the new feature at a system level
* Nit change
* Change Period parameter for WarmUpPeriod parameter
- Change regression test to check if the new parameter keep backwards compatibility with indicators that do not set WarmUpPeriod
* Documentation change
* Fix tests bugs
- In CommonIndicatorTests.cs before finish the test it checks the period.value with the number of samples but for default the period.value was set to -1
* Change names
* Change WarmUp and RegisterIndicator methods
- Lean WarmUp indicator skip custom python indicators that don't define WarmUpPeriod parameter
* Call WarmUpIndicator manually
- Add a new "bridge" method called WarmUpIndicator in QCAlgorithm.Python.cs to set up everything to call WarmUpIndicator in QCAlgorithm.Indicators.cs
- Change the regression algorithm to warm up the indicators manually
* Remove unnecessary code and add more tests
* Nit change
* Revert "Nit change"
This reverts commit da411f59c9.
* Fix bugs
* Try fix bugs
* Add C# regression test
- More nit changes
- Fix bugs
* Requested changes
* Remove unnecessary code
* Requested changes
* Nit changes
- Add new Python class to check a custom indicator, which doesn't inherits from PythonIndicator, warms up properly
* Reduce redundant code
* Fix bug and add more unit and regression tests
* - Add more unit tests
* Nit change
* Test cleanup
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Remove unnecessary attribute `_period` in MarketProfile
- Set the value of WarmUpPeriod to the period in MarketProfile constructor
- Fix unit test to check VolumeProfile and TimeProfile indicators are being warmed up properly and its WarmUpPeriod parameter isn't zero
- If available, CoinApi converter will read previous date files to
rescue initial midnight ticks for the processing date. Also make sure
to drop ticks from tomorrow
* Implement IIndicatorWarmUpPeriodProvider
- Implement IIndicatorWarmUpPeriodProvider in PythonIndicator.cs
- Make a unit test to check whether the WarmUpPeriod is working as expected
- Make a regression test to check the new feature at a system level
* Nit change
* Change Period parameter for WarmUpPeriod parameter
- Change regression test to check if the new parameter keep backwards compatibility with indicators that do not set WarmUpPeriod
* Documentation change
* Fix tests bugs
- In CommonIndicatorTests.cs before finish the test it checks the period.value with the number of samples but for default the period.value was set to -1
* Change names
* Change WarmUp and RegisterIndicator methods
- Lean WarmUp indicator skip custom python indicators that don't define WarmUpPeriod parameter
* Call WarmUpIndicator manually
- Add a new "bridge" method called WarmUpIndicator in QCAlgorithm.Python.cs to set up everything to call WarmUpIndicator in QCAlgorithm.Indicators.cs
- Change the regression algorithm to warm up the indicators manually
* Remove unnecessary code and add more tests
* Nit change
* Revert "Nit change"
This reverts commit da411f59c9.
* Fix bugs
* Try fix bugs
* Add C# regression test
- More nit changes
- Fix bugs
* Requested changes
* Remove unnecessary code
* Requested changes
* Nit changes
- Add new Python class to check a custom indicator, which doesn't inherits from PythonIndicator, warms up properly
* Reduce redundant code
* Refactor error message for reproducability
* WIP Refactor GetMaximumOrderQuantityForTargetBuyingPower
* Additional tweaks and tests
* Address shorted margin case
* Reinstate tests with new modified function
* Update regression
* Some cleanup
* Expand test set
* Refactor solution
* Address review
* Final adjustments and cleanup
* Expand error message for GetAmountToOrder for safety and reproducibility
* Address reviews
* Add base class for pivotpointshighlow
* abstracts do not work well with python switch to virtuals and use pivotpointhighlow as base implementation
* added missing documentatino header
* moved reset back to original location
- Implement IIndicatorWarmUpPeriodProvider interface in both indicators
- Add test to check the new functionality is working as expected. To test WindowIndicator IIndicatorWarmUpPeriodProvider there was used WindowIdentity indicator tests because that indicator only inherits from it
* Update SPDB with price magnifier parameter
- Add a new column for new parameter in symbol-properties-database.csv
- Change InteractiveBrokersBrokerage.cs implementation of GetContractPriceMagnifier() method in order to get the parameter directly from the SPDB
- Add test in SymbolPropertiesDatabaseTests.cs to check if the new parameter is loading properly
* Change GetContractPriceMagnifier() method name and implementation
- Change NormalizePriceToLean() and NormalizePriceToBrokerage() methods
* Nit change in GetSymbolPriceMagnifier() method
* Remove GetSymbolPriceMagnifier() method
- Remove redundant and unnecessary parameters in NormalizePriceToLean() and NormalizePriceToBrokerage()
* Requested changes and SPDB update
- Update SPDB with minimum order size parameter for futures in cents. The minimum order size parameter was extracted from the CME group API
* Address reviews. Adding unit test for failing case
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* fix exchangeinfo downloader
* normalize data, update symbol properties
* min ordersize should be in base currency
* check min order size using quote currency price
* check order size in quote currency
* improve checks
* improve code; more unit tests
* more description
* fix failing test
* Move Aggregate Quote and Trade Bars to utils
* Add decimal and long overload to UnixTimeStampToDateTime
* Add Kraken OrderType check in CanSubmitOrder
* PR !5969 review fixes
* Implement scheduled event sampling solution
* Use UTC time, only update daily portfolio value once a day
* For daily resolutions sample chart always
* Cleanup
* Drop resample daily all together
* Force final sample
* Regression updates
* FIx LiveResultHandler to update portfolio and benchmark values outside of sampling event
* Name the daily sampling event
* Address review pt 1
* Drop force and use reference wrapper
* Adjust tests
* Fix warning for Benchmark Timezone Misalignment and also add test
* Fix for daily resolution orders and test adjustments
* Also warn on universe settings with daily resolution
* Update missed regression
* Fix reference wrapper use
* Update regression after rebase
* Add values back in for Daylight Algo
* Have statistics builder skip day 1 performance
* Regression adjustments
* Test adjustments
* Update regression unit test
* Adjust some regressions starts to show performance values
* Add hourly algorithm for beta comparison
* Address missing Python regression changes
* Remove null comment
* Bitfinex Fee adjustment
- Fix for bitfinex fee adjustment for cash account type only happen if
the currencies are the expected ones, else log message.
* Adjust error handling to terminate algorithm
* Add bitfinex TESTBTCTESTUSD
* Minor improvements
- Add bitfinex test symbols to the SPDB
- Make SecurityDefinitionSymbolResolver log using trace
- Minor tweak for Bitfinex DQH lock contention
* - Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type
- Change CanSubmitOrder() method implementation in GDAXBrokerageModel.cs to get the MinimumOrderSize from the security SymbolProperties directly
- Add CanSubmitOrder() method in Binance, Bitfinex and Kraken Brokerages models
- Add tests to check if CanSubmitOrder() method is working as expected in Binance, Bitfinex, Kraken and GDAX brokerages models
- Add tests to check if MinimumOrderSize parameter is loading correctly from symbol-properties-database.csv
* Revert "- Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type"
This reverts commit e0fd66a5fa.
* Revert "Revert "- Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type""
This reverts commit b0f37935fc.
* Requested changes in Crypto brokerage models
- Remove unnecessary commas in symbols-properties-database.csv
- Add TestHelpers class to reuse GetSecurity() method
- Add IsValidOrderSize() method in DefaultBrokerageModel class
- Remove unnecessary classes for brokerage models test helpers
* Nit change
- Remove unnecessary attribute from DefaultBrokerageModel.cs
* Nit change
* Nit changes and Regression test
- Add Bitfinex BTCUSD hourly data for 2 days
- Add regression test BuyBTCWithLessThanOneDollarAlgorithm.cs to test that the order size is taking into account when placing or updating an order
- Nit changes
* Change CanUpdateOrder() implementation
- CanUpdateOrder() now checks if the requested quantity value is valid
* Requested Changes
- Change regression algorithm BuyBTCWithLessThanOneDollar.cs name and implementation
* Nit changes
* Add BaseDataCollectionAggregatorReader
- Add enumerable for BaseDataCollection
- Add BaseDataCollectionAggregatorReader
- Clean up CollectionSubscriptionDataSourceReader
* Address reviews
- Add new FileFormat that will be handled by the new Reader
- Adding unit tests for the new BaseDataCollectionAggregatorReader
- Some DataSourceReader duplication cleaning up
* Minor adjustment after some live trading tests
Adds Warnings to MOO and MOO using Tick Data
These warnings are meant to let the user know that it was not possible to find ticks marked with the `OfficialOpen`, `OfficialClose`, `OpeningPrints` or `ClosingPrints` flags.
* Fixes Market On Close Fill of Equity Fill Model
Only use trade data (`Tick` with `TickTrade` type or `TradeBar`) to get the closing price, since MOC is filled with the closing price.
- Fix unit tests to show that the new implementation only fills with trade data from the current open market.
- Change regression tests to reflect the bug fix. All other changes are the consequence of using `TradeBar` instead of `QuoteBar`.
* Address Peer-Review
Fixes the logic with tick data: looking for the OfficalClose or ClosingPrints only apply if data from the extended market hours is included.
Adds additional logic to handle missing trade data.
* Refactors Timeout Logic and Adds It to MarketOnOpenFIll
-Refactors timeout logic to address peer-review and adds it to MOO fill too.
- Adds unit tests
* Fix IB futures prices where PriceMagnifier != 1
* Fix deadlock in GetOpenOrdersInternal by moving ConvertOrder call out of event handler;
Minor refactor to combine NormalizePrice and ConvertPrice methods
* IB future and future options price magnifier
* Address review. IB API clean up
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Lean exchanges improvements
- Adding new Exchange class to avoid exchange code clash.
Adding and updating unit test
* Add Market for MapFile API
* Self review
* Rename Renko to ClassicRenko
* Rename WickedRenko to Renko
* Adjust tests and usages
* Backwards support WickedRenkoConsolidator
* Renames for consistency
* Refactor for two types in composite indicator
* Fixes for unit tests
* Non generic indicator base
* Further seperate non-generic indicator base and generic input base
* Adjustments
* Remove generic CompositeIndicator
* Add test suite
* Make each method an individual case
* Address review
* Testing cleanup
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Adds support for getting a Lean symbol based on FIGI, CUSIP, ISIN, SEDOL
* Address review: bug fixes for mapping, improve docs, cleanup & refactor
* Make LocalZipMapFileProvider and LocalZipFactorFileProvider only
initialize if they haven't been initialized yet.
* Address review: makes ISIN, SEDOL, CUSIP case-insensitive
* Cleans up map file provider in SecurityDefinitionSymbolResolver
* Modifies some test cases to test for case-insensitivity
* Add null check for SecurityDefinitions
* Fix unit tests
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add ValueAreaVolumePercentage Parameter
Remove hard coding of parameter and set it to optional parameter for the user.
* Update TimeProfile.cs to include valueAreaVolumePercentage parameter
* Update TimeProfile definition in QCAlgorithm.Indicators.cs
* Create Market Profile indicator.
* - Requested changes in MarketProfile indicator made it.
- TimeProfile and VolumeProfile created and tested.
- tp_datatest.csv and vp_datatest.csv extracted from https://github.com/bfolkens/py-market-profile
* Final potential Market Profile Indicator.
All unit test made it and the indicator is passing all of them
Test cases made it with python library from GH issue
* Code styling request changes
* Minor suggestions
* Minor renaming tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Few improvements
- Alpha Streams algorithm will add target securities right away
- If algorithm is warming up PortfolioTargetCollection will emit not
insight
- Fix local disk factor file provider to use local cache instead of data
folder
- EWAS PCM will not emit targets for securities which have not been
added by the algorithm yet, to avoid runtime exception
* Fix GetLastKnownPrice default order adding more unit tests
As most brokers use separate exchange codes for CBOE and CFE, we added the new Market.CFE (previously Market.CBOE was used for both exchanges). This change also avoids hacky code in symbol mappers.
* Fix zerodha async order event
consider fillQuantity as total filled quantity
cover edge case of fully filled
fix async events received from zerodha
indentation fix
* Fix coding style
* keep order in only CachedOrderIDs
* Alpha Streams improvements
- Warmup added securities so we can trigger an order right away
- Fix bug where AddSecurity returned a security which was not the one
being used
- Fix bug where EWAS PCM would remove from a dictionary while iterating
over it
* Fix unit tests
* Add QCAlgorithm.GetLastKnownPrices API
- Add new GetLastKnownPrices that will return the last known data point
for all subscribed data types.
- Fix for MHDB GetDataTimeZone which was using an invalid entry key
format for custom data.
- Fix for using GetLastKnownPrice/s to seed a security during creation,
when it's not added in the Algorithm.Securities collection
* Adding more unit tests for GetLastKnownPrice
* Address reviews
- Refactor GetLastKnownPrice/s to perform a single history request for
all data types
- After https://github.com/QuantConnect/Lean/pull/5872 trading API
changes numpy float64 was not converted correctly by pythonNet and
used an int. Reverting API changes and adding regression test. This
should be fixed at pythonNet layer
* Adds ETF(...) to UniverseDefinitions
* Adds ETF constituents universe framework regression algorithm
for C#/Python
* Address review: adds test cases for ticker/Symbol ETF universe additions
* Fixes bug where null Market would result in null dereference exception
* Address review: add missing Index tests
* Address review: don't hardcode market when creating constituent universe
* Uses Brokerage Model's default markets collection to determine
the market for the given security type
* Address review: restore QC500 and DollarVolume.Top(...)
* Restores algorithms related to both helper universe
definition methods
* Address review: remove copy to output directory for python algos
* Add example algorithms for ETF constituent universes using custom RSI alpha model
* Address review: adjust algorithm to use cache + algo RSI & clean up code
* Address review: make ETF Constituent RSI Alpha Model algo a regression test
* Address review: increase trade count and remove single trade logic
* Order handling improvements
- Execution model will only trigger market order if they are above the
minimum order margin portfolio percetage value
- SecurityCache.Reset is complete
* Python Import fixes
- Add regression test for ImmediateExecutionModel minimum order margin
check
* Add basic template Atreyu algorithm
- Add C# and Python basic template atreyu algorithm. Show casing how to
specify exchange to execute in different ways.
- Adjust trading API to allow specifying order properties to use
* Lean Exchange improvements
- Rename PrimaryExchange to Exchange
- OrderPropeties will use Exchange enum instead of string
- Adding BSE exchange value
* Regression tests fixes
* To fix the referenced issue
* Fixing up TrailingStopRiskManagementModel
* Fix-ups to address review
* Impl. tests + relevant improvements
* Minor logic improvement at first dictionary update
* Regression test small fix
* Adds explicit 'D' suffix for numbers in test case double arr
* Use integer values in test cases
* Removes failing testcases in MaximumDrawdownPercentPerSecurityTests (!) & renaming
* Revert "Removes failing testcases in MaximumDrawdownPercentPerSecurityTests (!) & renaming"
This reverts commit f9cd279f8c.
* Fix up for failing test cases
* Alpha holdings state
- Alpha result packet will optionally provide the algorithms portfolio
state
* Rename
* Convert AlphaStreamsPortfolio to data source
* Improvements on AlphaStreams algorithm
* Fix regression tests
* Add unit tests for EW AS PCM and fixing bugs
* Protobuf AlphaStreamsPortfoliot staState
- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM
* Add alpha license to Organization response
* Improvements EW AS PCM respects free portfolio value
* Fixes
- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType
* Fixed and adding more regression tests
- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
types
- Add support and regression test for algorithm alpha consumer with existing holdings
* Add AlphaStreamsOrderEvent data type
* More logging for BinanceRestApiClient
* Changing LogLevel to Debug for history providers : Binance, Polygon
* Minor fixes
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Adds support for ETF constituent universes
* Adds filtering for universe data if it doesn't match the
universe subscription type
* Update mapping for ALL underlying Symbols if
`Symbol.UpdateMappedSymbol(...)` is called. Required to support
constituent ETF universes that might have mapping events
* Delistings of composite constituent universe Symbol will result in
removal of universe securities.
* Added regression algorithms for ETF constituent mappings (C#/Python),
along with data required to run locally
* Refactor universe delistings in SubscriptionSynchronizer -
big thank you to @Martin-Molinero :)
* Address review: update regression algorithms and add explanatory comments
* Address review: add additional checks to delisting regression algorithms
* Adds new regression algorithm testing the addition of a universe
without calling AddEquity() and asserts same behavior
* Fix SubscriptionDataReader auxiliary data out of order
- Fix for SubscriptionDataReader emitting auxiliary data out of order
due to sparse that and enumerator refresh logic. Adding regression
algorithm
- Minor tweaks for unit tests failing on and off
* Fix for SDR internal subscriptions
- Fix for SubscriptionDataReader enumerator refresh for internal
subscriptions which were ignored. Adding regression test
- Updating custom data regression algorithms affected by the issue
- DropboxBaseDataUniverse was emitting a custom data point being end time
- UnlinkedTraderBarIconicType was emitting a single data point
of the underlying SPY minute data when if should of emitted
all data points
* websocket-refactor for websocket message as text and binary
Initial commit
websocket support for text and binary messageData
code style fixes
test fix
* delete un-used file
* zerodha websocket textMessage parse
* messageData class abscration
* Empty
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix IB history requests
- Fix time zone bug causing incomplete results
- Use RegularTradingHours flag only for equities
- Second resolution data is now requested in 30 minute chunks (instead of 1 minute):
https://interactivebrokers.github.io/tws-api/historical_limitations.html
* Filter history result respecting market hours and IncludeExtendedMarketHours flag
* Add history unit tests
* Improve python exception parsing
- Improve python exception parsing adding support for line shift. Adding
unit tests
* PythonException revert change
* Centralized and normalize algorithm runtime handling
* Adding support for C# line and file exception report
* Fixes Market On Open Fill of Equity Fill Model
Only use trade data (Tick with Trade type or TradeBar) to get the open price, since MOO is filled with the opening action price. Ensure that this method doesn't use trade data from before the market opens for high-resolution data case.
Fix unit tests to show that the new implementation only fills with trade data from the current open market.
Change regression tests to reflect the bug fix.
In the `ExtendedMarketHoursHistoryRegressionAlgorithm`, MOO was filled with extended market hours.
* Fix Bug for Tick Resolution Case
For tick susbcription, the tick with the open price information is the the first valid (non-zero) tick of trade type from an open market.
Addresses peer-review by moving the if-condition for data belonging to the open market where the subscriscribed types are checked.
* Fix Bug for Low Resolution Edge Case
For the edge case where the order is placed after the trade bar is open, for example, order places at 1 pm with daily-resolution data. The fill model will not use the open of the bar that will close at midnight, since this value is prior to the order.
Adds unit test.
Change regression tests to reflect the bug fix.
In the `RegressionAlgorithm`, MOO was filled with open prior to the order. The algorithm now has one order less, since the last MOO would need to wait another day to be filled.
* Implements SaleCondition and Exchange Check For Tick
- Adds additional unit tests for MOO
* Fixes Regression Test in DataConsolidatorPythonWrapperTests
* Addresses Peer-Review
- Adds new unit test cases.
* Update Lean Foundation docker files to IBGateway v985
* Add apt-get update for ARM foundation
* Update default value of IB gateway
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* WIP removal of custom data references & tests updates
* Regression algos updated and python algorithms moved to DataSource repos
* Fixes failing unit tests
* Add "LiveDataTypes" field to LiveNodePacket
* Adds Initialize() to IDataChannelProvider
* Adds new extension method to convert
HistoryRequest -> SubscriptionDataConfig
* Address review: Add protobuf definitions for Iconic data types
* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo
* Small adjustments to variable naming and documentation
* Move test files to respective DataSource repos
* Add support for IBGateway v985
- Effective in TWS version 985 and later, for US stocks the bid, ask, and last size quotes are shown in shares (not in lots).
* Add missing error log in IB symbol mapper
* Trigger build
- Create Lean common engine initialization class, which will be called
by research and Lean launcher
- Fix backtest deserialization bug due to failing to handle decimal
value
* Refactor pandas mapper to support newer versions
* Clean up remapper
* Finalize deprecated tests and adjust those that should apply to new pandas
* Move mapper to its own file
* Add supporting Py tests and setup instructions
* Add license
- Calling RemoveSecurity() will remove the security from all the
universes holdings it, there was a race condition here where it would
just take the first universe and remove it from it.
Adding regression test adding and removing an option contract and it's underlying
- Take lock to refresh client to avoid null reference
exception
- Reduce gate rate
- Use extension methods to await tasks
- Avoid copying the data multiple times
* Update BinanceBrokerage to handle more than 512 symbols
* Address review
- fetch symbol weights only if required
- remove code duplication
* Add rate limiting for new connections
* Update WebSocketMessage to include the websocket instance
* Handle resubscriptions on reconnect
* Address review
* Address review
* Remove unnecessary locking
* WebSocketClientWrapper updates
- remove allocation of receive buffer on each message
- add missing lock in Close method
- log message data when message type is Close
- fix race condition after unexpected websocket close
* Set WebSocketClientWrapper task to LongRunning
* Add missing check in GetHistory
* Fix exceptions with Binance downloader
- closes#5794
* Update BinanceBrokerage to handle more than 512 symbols
* Address review
- fetch symbol weights only if required
- remove code duplication
* Add rate limiting for new connections
* Update WebSocketMessage to include the websocket instance
* Handle resubscriptions on reconnect
* Address review
* Address review
* Remove unnecessary locking
* WebSocketClientWrapper updates
- remove allocation of receive buffer on each message
- add missing lock in Close method
- log message data when message type is Close
- fix race condition after unexpected websocket close
* Set WebSocketClientWrapper task to LongRunning
* Add missing check in GetHistory
* Fix exceptions with Binance downloader
- closes#5794
* Update Bitfinex symbols in symbol properties database
* Update BitfinexBrokerage to use BrokerageMultiWebSocketSubscriptionManager
* Address review
* Remove unnecessary locking
* Remove old channels on resubscription
* logging and kite connection retry
(cherry picked from commit 9845a2d79e86c338093cadcae7f41b29d897c78c)
* changed thread.sleep time for kite connection retries
* websresponse retry on error - coding style change
* datetome parse to convert time in utc
* indentation code-style fix
* zerodhaDownloader support for dataTimeZone
* removed code redundancy for single conversion without hardcoded TZ
* removed hardcode dataTimeZone and code refactor
* returns exchange TZ to lean and data TZ to local disk from downloader
* Kite historical candle timestamp TZ conversion
* added unit test for kite historical candle
* access modifier correection
* Filter out small orders based on Setting
- BuyingPowerModel will filter out small orders based on algorithm
setting, a % of PTV, instead of hard coded 1 share value. Addin unit
and regression tests
- Updating regression algorithms to use new setting, reduce order trades
* Update regression algorithms
* zerodhaDownloader support for India Market
(cherry picked from commit c36272ef9c4c7eddfff77413d900faab3a2a8654)
* indenation styling fix
* zerodha donwloader
* Market.India support from zerodhaSymbolMapper
* code styling changes
* re adding KiteProductType
* check kiteProductType per order and get all Product Type Account Holdings
* use WithLockedStream and remove lock unlock stream
* filter holding if config productType set otherwise fetch all
* improvements in styling and properties
* unionbank has less margin requirements, second always works
- TransactionHandler will see invalid OrderEvent for an order which
tries to short an non shortable asset. Adding unit test, updating
regression algorithm.
- Fix Api data price path normalization, found while testing with
shortable provider. Adding unit tests
- Improve readme file, add nuget link, update Lean home page
- Increase 10min look back to 60 for setting currency conversion
- Replace error message for trace when minute resolution history request
is empty
* Added Augen Price Spike Files
* AugenPriceSpike Update
Adjusted indicator calculations and added test data from Trading View
* Requested Changes
* Minor changes
* Add reproducing unit test
* Apply fix to force adjustment if needed
* Adjust fee to correctly replicate original issue
* Adjust solution
* Update unit test to reflect the way this function is used
* nit - rename test
* Cleanup error message to improve reproducibility
* nit - Drop unneeded var
* Add in adjustment test cases as well, fixed
* Fix minor race condition in brokerage message handler
- Fix race condition where a message could be enqueued and
wait for a new call, not being processed ASAP
* Reduce BrokerageTransactionHandler logs
* Fix coarse universe generator on single processor
* Make sure MaxDegreeOfParallelism value is not less than 1
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add BrokerageConcurrentMessageHandler
- Add Brokerage helper `BrokerageConcurrentMessageHandler` class to lock message stream
while executing an action, for example placing an order, adding unit tests.
- Binance will use new message handler
* Fix unit test
* Implement solution with lightest changes possible
* Update regressions v1
* Adjust AlgorithmTradingTests
* Adjust PatternDayTradingMarginBuyingPowerModel tests
* Drop need to loop twice
* Adjust last unit test, with calculations included
* nit - comment fix
* Break out adjustment calculation to static function; add unit test
* Update Py regression
* Upgrade adjustment calculation to be smart enough to get us to target always
* nit - cleanup GetAmountToOrder
* Add license to test
* nit - comment fix
* cleanup GetAmountToOrder further
* Add additional test cases
* added symbolData class to store exchange and token values of tickers
* support for India Market
(cherry picked from commit 5f629dcb9e72426dc7b560e782cb1fae72861574)
* removed local vscode workspace file
* support for india market instead of nse and other exchanges
* changed hardcoded india values to Market.India
* changed zerodha getcash fromm available to net
* addded edge case handoing with expections
* support for market info (nse, bse) for india market
* exchange info for EmitQuoteTick
* basic testing with c#
* c# algo
* c# testing
* live orders during market open success
* support orderproperties for exchange value for order placement
* added exceptional handling cases
* fix xml descriptions for the files
* fix zerodha-product-type in DefaultOrderProperties
* add basic template for india market
* fix orderProperites for default values
* fix for local data, python implementation not available
* fix access modifiers of SymbolData
* fix variable name case
* fix single check for exchange, call to base
* fix styling, redundent code and access modifers
* fix non static not callable from static
* fix static
* Adjust AlgoSeek Futures price multipliers to quote prices in USD
* Revert changes to AUP scaling
* After a second review, seems that this contract isn't scaled.
The contract multiplier differs from the NYMEX Aluminum Index's
contract multiplier, and so does the minimum price variation.
The minimum price fluctuation seems consistent with CME's
quoted prices, which also claims the contract is quoted in USD.
* Fix futures contract multipliers for some scaled contracts
- Add new CompositeDataProvider that will wrap multiple instance. Allows
using for example the ApiDataProvider and a DownloaderDataProvider.
- Adjusting ProductJsonConverter after recent Modules addition
* Add new DownloaderDataProvider
- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
IBrokerage implementation as data source
* Add support for downloading symbol chains data
The Kraken allow queries from pairs of format XBTUSD and XXBTZUSD but it will only return data with the XXBTZUSD causing errors when XBTUSD ticker is specified.
This commit adds a more meaningful error when KDL is called with XBTUSD format.
* Update IBAutomater to v2.0.39
* Downgrade IB logging level from Detail (5) to Information (4)
- The online documentation warns of performance overhead when setting logLevel=5
* Apply fix for both start and end month cases
* Add unit tests for both cases
* Fix weekstart/weekend for same issue; add unit tests
* nit - rename tests
* Adjust logic
* Update charts that are not empty and not default values
* Testing
* Filter final results
* Explicitly filter alpha charts if not needed
* Use filtered charts for Live results handler
* Refactor solution to manage alpha charts internally by creator
* nit comment fixes
* Implement drawdown as default chart
* Implement Capacity Estimate for backtest default chart
* Add default values to QC.Plotting
* Adjust drawdown calculation
* Only sample "Capacity" once a day
* Implement Volume chart
* Nit fixes
* Volume Chart adjustments and fixes
* Rename to "Assets Sales Volume"
* Implement Exposure Sampling
* Store algorithm currency symbol, round capacity to nearest 1k
* Add new plots to default charts
* nit change name
* Address reviews
* Reduce duplication and clean up exposure sampling
* Address reviews
* Improve sample exposure
* Address review
* Only enumerate holdings once
* nit - comments
* Post rebase fix
* Don't need to round anymore
* SymbolJsonConverter improvements
- SymbolJsonConverter will be able to deserialize symbols expressed in
user friendly format. Adding unit test
- Adding a couple of helper methods, removing duplicated code
* Fix FOPs expiration year parsing
* Add IResultHandler.BrokerageMessage
- Result handlers can receive notifications for brokerage message events
* Make LiveTradingResultHandler.RuntimeError virtual
- Reduce thread contention in coinApi DQH by taking a lock per symbol
versus a global lock.
- Improve tick creating by reducing object being wasted on empty
constructor
* Updates Equity Market Data
* Updates Unit Tests
* Updates Regression Tests
In this commit we include regression tests with small changes (slightly different CAGR, Alpha, etc, but same number of trades) due to the data update.
* Updates Regression Tests 2
The following regression tests were adapt because of verification of hard-coded market data values:
- `AdjustedVolumeRegressionAlgorithm`
- `HistoryWithSymbolChangesRegressionAlgorithm`
- `OptionRenameRegressionAlgorithm`
- `RawDataRegressionAlgorithm`
- `SwitchDataModeRegressionAlgorithm`
The following regression tests have more trades since adjusted prices allowed more 1-2 shares trades that were rounded down to zero before:
- `AddUniverseSelectionModelCoarseAlgorithm` 23 -> 35
- `MeanVarianceOptimizationFrameworkAlgorithm` 12 -> 14
- `PortfolioRebalanceOnDateRulesRegressionAlgorithm` 298 -> 324
- `PortfolioRebalanceOnInsightChangesRegressionAlgorithm` 83 -> 86
- `ScheduledUniverseSelectionModelRegressionAlgorithm` 86 -> 90
- `SectorExposureRiskFrameworkAlgorithm` 17 -> 22
- `SetHoldingsMultipleTargetsRegressionAlgorithm` 8 -> 9
- `StandardDeviationExecutionModelRegressionAlgorithm` 196 -> 199
- `UniverseUnchangedRegressionAlgorithm` 11 -> 17
- `VolumeWeightedAveragePriceExecutionModelRegressionAlgorithm` 237 -> 238
Especial cases:
- `BlackLittermanPortfolioOptimizationFrameworkAlgorithm` 18 -> 17
- BLM model sensibility
- `OptionChainedAndUniverseSelectionRegressionAlgorithm`
The following regression tests have different Capacity because of different volume from lowest capacity asset, except:
- `OptionEquityCoveredCallRegressionAlgorithm` New lowest capacity asset is underlying
- `OptionEquityCoveredPutRegressionAlgorithm` New lowest capacity asset is underlying
* Revert File Update for SPWR and SPWRA
* Fix Regression Tests
Temporarily removes python regression test for `MeanVarianceOptimizationFrameworkAlgorithm` as the `MeanVarianceOptimizationPortfolioConstructionModel` for each version are yeilding different results. If we use C# version in `MeanVarianceOptimizationPortfolioConstructionModel.py`, the results match.
* Changes Optimization Method in MinimumVariancePortfolioOptimizer [Py]
Uses `trust-constr` method.
See https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html
* Use Python algorithms directly in their location; also refactor pythonpath additions to one method
* Update VSCode Launch options
* Drop issue these changes fixes
* Use AddRange
* Python research import improvements
- Improve start.py for research env
- Remove unrequired imports
* Centralize algorithm imports
* Add regression test GH action
* Unit test python import clean up
* Join research and main imports
* More python import clean up
* Fix failing skipped regression algorithm
* Stop update when properties will fail to update
* Only enforce our case after the BB is ready
* Nit comment fixes
* Add unit test
* Use faster DateTime.UtcNow
* Updates to default notebooks
* Update readme
* Notebook adjustments
* Drop launch script support
* Use our cwd if no QC dlls are in AppDomain directory
* Modify Intialize.csx to use parent directory only if QC files are detected there
* Clean up some details after testing
* Adjustments
* nit, NullOrWhiteSpace
* Enable loading live virtual position tracking
- Enable GDAX paper trading configurations
- GDAX will optionally load existing holdings from live job package
- Bitfinex will try loading existing holdings from live job package for
cash account type
* Binance live holdings loading
* Adds VIXCentral Contango data
* Refactor cached alternative data in live mode to
use web endpoint if the data is not available locally
* Remove fallback to cache bucket for CBOE/FRED/EIA/VIXCentralContango
* Address review: force VIX as ticker for all VIXCentralContango data
* Normalize live holdings initialization
* PaperBrokerage config based live state recovery
- Increase algorithm status storing sampling period
- Fix bug where duplicate cash entries could happen
* Applies a fix
* Fixing the tests to cover downloader func. Fixing the queue handler
* this has to be removed
* -- Fixes -- [WIP]
* Implements the way to upload historical bars in async fashion. [ WIP ]
* Finishes GetEquityTradeTicks() impl.
* Do not break iteration if result is null
* Uses DisposeSafely and fixes a typo
* To use IEnumerable for return type in GetHistoryAsync
* Fixes GetEquityQuoteTicks()
* Removes GetHistoryAsync() and reverts the associated with it changes
* SubscriptionManager -> revert the changes
* PolygonDataQueueHandler -> use SIP timestamp as it keeps the order
* PolygonDataQueueHandler -> use Log.Trace for the no data {date} message
* Fixes GetForexQuoteTicks()
* Introduce ResponseSizeLimitAggregateData
* Fixes up GetCryptoTradeTicks()
* Small fixes for tick processing methods
* Removes launchSettings.json
* Refactor download data logic
- Replace WebClient for HttpClient
- Remove duplicated download logic and parsing for a single static
method
* Update PolygonDataQueueHandler.cs
Remove commented code
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Use DataProvider for fetching Shortable and FutureMargins files
* Use data provider for AllShortableSymbols
* Refactor of NeedToDownload logic
* Adjustments
* nit - cleanup
* Expand test cases
* Create the directory if it doesn't exist
* Add immediate logging for IB brokerage unit tests
* Filter brokerage message event for IB warning
- 10197 - No market data during competing live session.
* Update IBAutomater to v2.0.32
* Update IBAutomater to v2.0.33
* Use determined quantity for exercise order
* Add regression algorithm for issue
* Minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Ensure data is downloaded only once with the ApiDataProvider. Adding
unit test
- Log invalid securityType ApiDataProvider requests once
- Assert Factor file and map file providers are compatible with the
ApiDataProvider
* Use lean data key as param for request
* key -> filePath rename and some cleanup
* Refactor
* Add Organizations Endpoints
* Add some organization api wrapper objects
* Address namespace issue
* Reorganize Api Test into seperate files using one ApiTestBase
* Add Organization tests
* Use capitalized "API" test namespace to reduce amount of file changes
* Add License to test base
* Update /data endpoint functions and response objects
* Update ApiDataProvider Logic
* Handle deserialization of organization products
* Simplify converter
* Only throw for equity requests when not subscribed to map/factor files
* Add missing header
* Make arguement exception
* Api adjustments
* Add Zip factor and map file providers
- Common project will now reference Compression project and not the other way
round.
- Adding Zip FactorFile and MapFile providers
* Refactor FactorFileProvider to use DataProvider to fetch files
* Use resulting MinimumDate in construction of FactorFile
* Nit FactorFile comments and arrangement
* Refactor MapFileProviders to use DataProvider for fetching files
* Refactor ZipFileProvider
* Clean up
* Refactor Backtesting Future/Option chain providers to use dataprovider
* Fixes for data/ endpoints and test adjustments
* Response objects adjustments/cleanups
* ApiDateProvider fixes and testing
* Add LocalZipFactorFileTests
* Update ApiDataProvider download test to verify stream is not null
* Implement posting of agreement summary and signed time
* Mark all Api related tests as explicit and document details on running
* Clarify default token on ApiTestBase
* Adjust summary
* Update Api responses for QCC, except org products which are sold in USD
* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests
* Fix multiple markets for ZipFactorFile provider
* Use Symbol as cache key
* Api.cs review
* Dispose of factorFileStream after reading
* Use zip.EntryFileNames
* Address a few reviews
* Few more fixes
* Address Api Review
* Add Job Org id to config
* Minor tweaks
* Compare with invariant culture
* Fixes Option Universe selection
* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider
* Fix research
* Fix null reference exception
* Make duplicate log debug
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fixes bug where raw data was incorrectly be filtered when ID was duped
* Filter CoinAPI raw data by distinct Symbol created from the SymbolMapper
* Fixes bug where only quotes or trades would be processed, not both
* Trigger rebuild
* Attempt to remove SecurityInitializer from UniverseSelection stack entirely
* Fix broken Python Universe Selection model
* Remove mistake change (meant for another branch)
* FOPs margin as a curve
- Adjust FOPs margin requirement so that it behaves like an 'S' curve
for the contract strike, around the underlying price and it's margin requirement.
Based on IB behavior. Adding unit tests for different future contracts
- Improve a couple of unit test cleanup
* Add ES Short Call unit test
* Adds filtering of negative ticks in CoinAPI converter
* Tick data will remain the same, only aggregated data
will have changes reflected in its output as a result
* Adds suspicious tick filtering to ticks
* Address review: Add Crypto Trade Tick suspicious test case
* Implements PivotPointsHighLow indicator + tests
# Conflicts:
# Tests/QuantConnect.Tests.csproj
* Some fixes
* Refactoring + Adds NewPivotPointFormed event
* Fixing tests
* Some fixes for the Get methods to not throw when points array is empty
* To address review + more fixes
* Fixing xml comments typos
* Implements separate rolling windows to calculate highs and low
* Makes the number of last stored indicator values as an input parameter
* Overrides Reset()
* Adds a helper method
* Change numerical return to Enum types
* Changes IsReady condition :
the indicator is ready and starts calculating the pivot point when any of the rollings is ready
* Address reviews
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Update launch options and tasks
* Update readme
* Mention building manually is optional
* Add portable pdbs to use C# coreclr debugging
* Update VS and VSCode docs
* Add reference to Lean CLI in main readme
* remove run_docker scripts
* Remove PyCharm files
* Cleanup
* Remove run_docker tasks from VS Code
* Address typos
* Implemented AlphaVantageDataDownloader
* Style and documentation changes requested
* Throw exception for unexpected response content
* Added rate limiting to AlphaVantageDataDownloader
* Minor tweaks and rebase
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Apply changes from #2146 on current master
Credit for most of these changes goes to https://github.com/viliwonka.
Rebasing three years of changes didn't go so well, so I manually
applied the changes in #2146 on the current master branch.
This commit attempts to represent the changes in #2146 as closely as
possible, no cleanup has been performed yet. Despite this, some changes
were inevitable because part of #2146 has been implemented another way
in the past three years.
* Clean up
* Process review comments
* Add tests
* symbol -> potentialConversionRateSymbol
* Ensure regression tests run/get data
* Fix broken test
* Process review comments
* Process review comments
* Simplify currency conversion logic
* Performance improvement, doc updates and test rename
* Rebase fixes
* Remove obsolete properties
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Minor tweaks
- Add AccountCurrency and symbol for live result packet
- Improve performance of Tick clone and reading
* Reduce unit test from 9s to 500ms
* Do not restrict unrequested securities by 1.0m leverage, let security initializer do its job
* Use UniverseSettings for leverage, fillforward, and extendedmarkethours
* Update unrequested security message
* Use UniverseSettings ExtendedMarketHours for underlying on OptionContracts
* Add unit test to prove setting persistence
* Cleanup unit test
* Adjust crypto unit test assert
Co-authored-by: Colton Sellers <Colton.R.Sellers@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Fix FutureOptionMarginModel Margin Requirements
- Fixes FuturesOptionsMarginModel margin requirements being zero due to
underlying being null on model creation. Adding unit test
* Address reviews
- SecurityService will set underlying when creating a new security when
provided
* Fixes SEC rate limiting issue by reducing clustering of requests
* We add a user-agent header to all outgoing requests for SEC
as they require it to identify the client, otherwise we're rate
limited.
* Some adjustments to how we wait for work to be done, and RateGate
rate of requests. We wait for all requests now/don't use tasks
anymore
* Address review - remove class HttpClient in favor of local instance
* Upgrade IBAutomater to v2.0.0
* Upgrade IBAutomater to v2.0.1
* Upgrade IBAutomater to v2.0.6
* Upgrade IBAutomater to v2.0.7
* Upgrade IBAutomater to v2.0.9
* Upgrade IBAutomater to v2.0.10
* Upgrade IBAutomater to v2.0.12
* Upgrade IBAutomater to v2.0.13
* Upgrade IBAutomater to v2.0.14
* Upgrade IBAutomater to v2.0.15
* Reset disconnected flag on IBGateway exit/restart
This bug was causing invalid algorithm termination after reconnection
* Fix unhandled exceptions in exit/restart event handlers
* Upgrade IBAutomater to v2.0.16
* Upgrade IBAutomater to v2.0.17
* Update testing OrderProvider to clone orders as BrokerageTransactionHandler
* Add immediate logging for brokerage unit tests
* Tradier brokerage updates
- Fixed NullReferenceException with zero-crossing orders
- Added a few missing null checks
- Fixed issues in error handling
- Added new unit tests for some common errors
* Adds processed data directory to read price data from
* Make coarse universe generator look at data directory before failing to find daily data
* Set coarse generator output of missing daily file to debug log
* Add CoarseUniverseGenerator logs
* Fixes 100 nanosecond increment lookahead bias when parsing large numbers
* Whenever we parse a number that is has precision greater than
DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
between [0, 1000), excluding numbers divisible by 100,
we will have leftover nanoseconds between [0, 100) nanoseconds, but
they won't be factored in to the DateTime calculation, since casting
to `long` only takes the integer component of the number, so we lose
the extra nanoseconds that came with the decimal, and time is set to
the "floored" value without those nanoseconds.
Since .NET `DateTime` type has a limitation of only being able
to represent time in increments of 100 nanoseconds, by not
considering the sub-100 nanoseconds, we introduce a look-ahead
bias of at most 100 nanoseconds/1 tick
* Misc adjustment to make method use `decimal` instead of `double`
for increased precision when parsing large numbers
* Changes CoinAPI data converter to support processing raw files in original directory structure and file name
* Removes Market requirement from CoinAPI data converter
* Remove timeout on decompression of raw AlgoSeek futures data
* Updates SEC downloader to use HttpClient where requests were failing
* For some unknown reason, valid requests to a valid URL were
failing when using WebClient. Changing our requester to
HttpClient fixes the issue, and enables us to leverage
async capabilities where applicable.
* Added fault tolerance to index file downloads, including a
rate limit in case we've been rate limited
* Further refactoring; catches 429 errors, adds missing rategate calls
* Replace all usage of WebClient, force retry for all failures
* Adds optional config value for Benzinga News API key in downloader
* Modifies Estimize Downloader api config name and fixes directory not found bug
* Refactor Estimize to speed up processing time
* Adds ticker limits if desired
* Misc. bug fixes, performance improvements, code cleanup
* Remove debug log statements leftover from previous commit
* Add support for non-tick Index resolutions in LeanDataWriter
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Lower requests/second for SEC downloader, add missing rategate call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Fix FOPs maintenance margin requirements since they were relaying on
the underlyings holdings cost to return 0 in some cases. Updating unit
tests
- Fix Option exercise order being rejected because of margin not
available, will now take into account the margin used by the option
itself. Adding unit test
* Add property for capacity. Remove unused variable
* Move SymbolCapacity and CapacityEstimate to common, passed through Symbol to runtime statistics
* Add null checks
* Remove uninvested and untradable assets from capacty calculations
* Add SymbolCapacity influential period
* Updates Regression Tests
- DelistingEventsAlgorithm
- Allows additional contributions from delisted AAA.1
- DelistingFutureOptionRegressionAlgorithm
- Removes DC01H12 contributions one month later
- FutureOptionBuySellCallIntradayRegressionAlgorithm
- Allows additional contributions from future after expiry replacing the contribution of the next contract option
- DelistedFutureLiquidateRegressionAlgorithm
- FutureOptionCallITMExpiryRegressionAlgorithm
- FutureOptionCallITMGreeksExpiryRegressionAlgorithm
- FutureOptionPutITMExpiryRegressionAlgorithm
- FutureOptionShortCallITMExpiryRegressionAlgorithm
- FutureOptionShortPutITMExpiryRegressionAlgorithm
- FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm
- Allows additional contributions from future after expiry
- FutureOptionCallOTMExpiryRegressionAlgorithm
- FutureOptionPutOTMExpiryRegressionAlgorithm
- FutureOptionShortPutOTMExpiryRegressionAlgorithm
- IndexOptionCallITMGreeksExpiryRegressionAlgorithm
- IndexOptionCallOTMExpiryRegressionAlgorithm
- IndexOptionShortCallOTMExpiryRegressionAlgorithm
- Allows additional contributions from option after expiry
- MACDTrendAlgorithm
- Removes contribution when SPY is not invested for over one month
- UniverseSelectionRegressionAlgorithm
- Allows additional contributions from delisted GOOAV replacing GOOG (new symbols)
* Adds Lowest Capacity Asset to Regression Tests
* Normalize expected value -0, because -0 is also written to file if updated
* Write Symbol.Value for lowestCapacitySymbol or empty string for empty Symbol
* Update Regressions
* Update 'Lowest Capacity Asset' to Symbol.ID
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Co-authored-by: Colton Sellers <Colton.R.Sellers@gmail.com>
* Updates Fundamental Data
* Updates Regression Algorithms
- `CoarseFundamentalTop3Algorithm`
- New coarse has higher `DollarVolume` for `FB`
- `CoarseNoLookAheadBiasAlgorithm`
- Change from `SPY` update not included in #5493
- `SectorExposureRiskFrameworkAlgorithm`
- `HasFundamentalData` was false for `GOOG` on 20140401 and 20140402.
* Updates Unit Tests
Minor changes in expected values
* Updates Regression Tests
Should have been included in #5493:
- `OptionsExpiredContractRegression`
- `FuturesExpiredContractRegression`
* Adjusted Quantity By Lot Size in OrderTargetsByMarginImpact
`OrderTargetsByMarginImpact` didn't calculate the order value with the quantity adjusted by order size which less to values that did not reflect the actual order value.
It has a particular affect in `SectorExposureRiskFrameworkAlgorithm` where the Python and C# versions have the same orders but placed in a different sequence because of decimal/double precision.
* Reduce dictionary access x3 on OrderTargetsByMarginImpact
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Redefine IsReady and WarmUpPeriod
* Update warm up periods of other indicators to account for ROC warmup change
* Update regressions affected by this change (because of HistoricalReturnsAlphaModel using ROC)
* Add simple unit test for issue #5491
* Cleanup warmup period math and add comments
Co-authored-by: Colton Sellers <Colton.R.Sellers@gmail.com>
* Reconcile and consolidate SetupHandlers
* Centralize GetConfiguredDataFeeds()
* nit - extra space
* Check for null, also allow null to be passed back if no config value
* Fix breaking test
* Cleanup
* fix return var
* remove unneeded if/else
* Minor changes
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Use UniverseSettings.DataNormalizationMode for securities added in Algorithm
* Stop SubscriptionUtils from forcing Adjusted mode
* Return behavior to original and add comments
* nit typo
* Add regression
* Add unit test that verifies DataNormalizationMode can be altered manually by security
* Cleanup and add Py version of regression
* Update projects to use .NET 5.0, the successor to .NET Core
* Fix ambiguous errors. Add IBAutomator net5
* Remove FXCM
* Upgrade IBAutomater to v1.0.51
ignored, and an empty message aborts the commit.
* Fix rebase
- Fix ambiguous Index
- Remove StrategyCapacity.cs
- Update System.Threading.Tasks.Extensionsy
* Remove unrequired references
* Fixes
- Travis will use dotnet, not nunit nor mono
- Remove mono from foundation image
- Fix python setup in research
- Fix unit tests
* Don't call ReadKey when input is redirected
* Fix ConsoleLeanOptimizer
* Research fixes
* Update comment
* Add vsdbg to Dockerfile
* Fixes
- Revert dockerfile FROM custom changes
- Adjust and fix regression algorithms
- Option assignment will be deterministic in the order
- 'Rolling Averaged Population' is calculated using doubles, updating
expected values.
- Update readme, removing references to mono
- Add missing Py.Gil lock
* Replace ICSharp with .NET Interactive
* Fixes after rebase
* CSharp research fixes
- Adding new Initialize.csx that pre loads all assemblies
- Adjusting template research file
- Moving steps in dockerfilejupyter
- Fix unit tests and regression tests after rebase
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Jasper van Merle <jaspervmerle@gmail.com>
* Add unit test
* Make test cover issue case
* Filter out times before startTimeUtc
* Refactor solution, fixes missing first date
* Fix case where none is expected
* Cleanup tests, add TriggerTimesNone
* Drop unused imports
* Add missing dipose call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Refactor HasSufficientBuyingPowerForOrder implementations
Adds Sufficient and Insufficient helper methods to HashSufficientbuyingPowerForOrderParameters
enabling syntax like:
return paraeeters.Sufficient()
returnparameters.Insufficient(reason)
The next change will add the initial margin required which will simply require
updating both of these helper methods to accept the value.
* IBuyingPowerModel: Add margin functions Maintenance/Initial/ForOrder
These were originally hidden in an effort to only expose what's necessary
for the engine to perform its work. Additionally, we encapsulated all of
the method arguments into parameters classes to prevent having to break
anyone in the future. Not including these foundational methods turns out to
be an oversight. These methods are not required by the engine, but rather by
other models. Another possible solution here is to add an additional abstraction
and include these methods on this new abstraction. BuyingPowerModel would then
explicitly implement these methods and models that depend on them would require
two code paths, one for when the buying power model implements this interface
and another for when it doesn't.
Tests were additionally updated to remove test model implementations created for
the sole purpose of exposing these private methods.
* Add ConstantBuyingPowerModel
Provides an implementation of IBuyingPowerModel that returns the same
constant value
* Update BuyingPowerModelPythonWrapper to use reflection for method names
Having a bunch of hard-coded strings is a sure fire way for someone to
overlook when changing methods. This change ensures that noone needs to
remember that this code exists :)
Cleans up the syntax around verifying a python object implements a particular
C# interface via the ValidateImplementationOf<T> method by having it return a
value since the only use cases are in constructors when setting the models.
I was initially going to update ALL python wrappers to validate the passed
in models, but such a change could break many things that are 'working' right
now. Such an effort should be saved for its own dedicated PR.
* Add Parameters/Result types for new buying power model methods
* Support computing maintenance margin for arbitrary quantities
The existing GetMaintenanceMargin function assumes that we're only interested
in the maintenance margin for the entirety of the provided security's holdings.
This makes it impossible to perform what-if analysis or to even ask how much
maintenance margin is devoted to a particular subset of the security's holdings.
This change adds the quantity to the MaintenanceMarginParameters class. Futures
and Options models also depend on holdings cost and holdings value, so they have
also been added to the parameters type. Finally, static factory methods were
added to improve discernment of intent: ForCurrentHoldings provides the existing
behavior and then ForQuantityAtCurrentPrice to support what-if scenarios where
we're looking for the change in maintenance margin if we were to execute an order
for the securiy at the current time step. Obviously a constructor is provided to
set all of the values explicitly, using any price metric the caller desires.
* Address review
- Fix BPM xml documentation
- Fix python unit tests and PythonWrapper validate method
* Add SecurityHolding.QuantityChanged event
Adding event handlers will allow us to orchestrate complex
events from distant parts of the codebase through wiring
them up. If we continue down this path, it will move us away
from the current, very 'mechanical' data flows expressed in
LEAN and towards a more modern, event processing based system.
This is but a baby step in that direction and the initial use
case is using this QuantityChanged event to trigger resolution
of the algoritm's positions groups. This is part of an effort
to improve the fidelity of options margin modeling where we'll
model an OptionStrategy as an IPositionGroup. This will allow
us to compute the margin requirements of an OptionStrategy as
a unit instead of computing margin of each security individually
in isolation.
See #4065
* PortfolioManager: Group fields and remove unused field
This codebase generally places fields as the first members, but
this class had some fields at the top, then some properties, and
then some more fields. This change brings all the fields together
at the top of the file and also removes pointless comments placed
directly above some of the fields. Additionally, an unused field
was removed.
* Remove unused _currencyConverter from Security
Looks like at some point the only code using this member variable was removed
and the necessary clean up was overlooked.
* Add Parse.Enum functions
* Support disabling regression algorithms by language via config.json
Adds 'regression-test-languages' to config.json and filters regerssion algorithms to
run based on this value. When cycling on a particular feature, it's nice to be able
to run the entire regression set while ignoring the python algorithms. Once the C#
algorithms are all passing, one can then go back and run C# and Python in a final run,
since 99% of feature work doesn't impact python specifically.
* Implement IComparable in SecurityIdentitfier
This can be used to deterministically sort securities and symbols
* Add .editorconfig to enforce common formatting for json/sh files
* Fix typo in IBuyingPowerModel.GetBuyingPower xml docs
* Add ListEquals/GetListHashCode and OrderDirection.Closes(PositionSide)
ListEquals and GetListHashCode are designed to be used together as they
complement each other according to C#'s requirements for Equals and
GetHashCode functions.
PositionSide.ToOrderDirection() extension simply converts a PositionSide
to its logical equivalent OrderDirection. Long->Buy, Short->Sell, None->Hold
OrderDirection.Closes(PositionSide) determines if a particular OrderDirection
would have the effect of reducing a position's absolute size. This function
greatly improves the readability of buying power functions that must provide
adjustments when an order/contemplated trade reduces/closes an existing position.
OrderDirection.Buy.Closes(PositionSide.Short)
OrderDirection.Sell.Closes(PositionSide.Long)
All other combinations return false
Adds ToArray/ToImmutableArray convenience functions that combine a call
to Select followed by To(Immutable)Array all in one function call.
* Add decimal.DiscretelyRoundBy extension method
Supports rounding a decimal value by an arbitrarily chosen maximum precision,
or 'quanta'
* Update FutureMarginBuyingPowerModelTests to respect the security's lot size
* Add core position group classes and abstractions
* Add initial/maintenance margin support, buying power model consistency tests
* Add SufficientBuyingPower and GetReservedBuyingPower to position group model
Includes update to BrokerageTransactionHandler to use position group BPM for
sufficient buying power checks.
* Resolve position groups on each fill
We need to update the state of our position groups on each fill so that
we can properly handle multiple orders within the same time step. We
also limit the number of positions sent into the resolver by removing
securities without any holdings.
* fixup! Add SufficientBuyingPower and GetReservedBuyingPower to position group model
* Add GetMaximumLotsFor{Target|Delta}BuyingPower
Instead of computing order quantity, these functions compute the
maximum number of position group lots, which is the position group
quantity, and is guaranteed to be a whole number, for the provided
target/delta buying power parameters.
The SecurityPositionGroupBuyingPowerModel delegates to the security's
IBuyingPowerModel by applying a scaling factor equal to the security's
lot size.
This change also updates references to IBuyingPowerModel.GetMaximum...
to use the new position group model methods.
* Convert remaining IBuyingPowerModel call sites to position groups
* Rename PositionManasger.CreateDefaultGroup -> GetOrCreateDefaultGroup
Better describes its behavior
* Add Position Groups readme.md
* Add Option Strategy BuyingPowerModel
- Adding CompositePrositionGroupResolver and
OptionStrategyPositionGroupResolver
- Adding OptionStrategyPositionGroupBuyingPowerModel handling option
strategies based on IBs margin table. Adding regression algorithms
- Few changes so that option strategies executed by multiple orders are
detected
- Adjust OptionStrategyDefinitionMatch to include equity legs in the
matching result
- Minor tweaks fixing previous rebase
- Minor fixes for existing option strategies definitions, adding new
missing strategies.
- Fixing minor bugs in option strategy matcher. Adding more unit tests
* Address self reviews
- Fixing bug in 'PositionGroupCollection'
- Few minor simplificaitons
- Adding BasicTemplateOptionEquityStrategyAlgorithm
* Address reviews
- Improve regression algorithms margin remaining and used assert logic to be exact. Taking into account spread and fees
Co-authored-by: Michael Handschuh <mhandschuh@gmail.com>
* Refactor to queue up start date changes to log on dispose
* Fix typo in warning collection used
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Collect and log symbols that had start date adjusted because of factor files
* Only post the message if the set has values
* Modify message to be [Symbol, Date] combo for more information on log
* Set a hard limit on the warning set to keep it from growing unnecessarily large
* Enforce a hard limit and improve message if full
* Enforce limit in log message because of possible threading adding past max
* Cleanup
* Log security mode change once per universe addition process
* Limit max size of the warning queue to 10
* Only allow to emit once per backtest
* Set limit as var, if at limit suggest more
* Always suggest more warnings may exist because we opted to log only once
* Improves greeks configurability and defaults for all option asset types
* Makes `StandardDeviationOfReturns` configurable by users, so that
greeks can be loaded according to user expectations and the series
of returns that they'd like to compute for `n` periods and timespan
of `T`, as well as resolution of the data in live mode.
* Changes resolution to max resolution available for the default
volatility model created for the security. Usually this only applies
to live mode, but if creating an instance of the
`StandardDeviationOfReturns` volatility model and no `updateFrequency`
is provided, the resolution's time span will be used as the default
value. Backwards compatibility for equities is maintained.
* Changes defaults for `StandardDeviationOfReturnsVolatilityModel`
to warmup greeks faster for other derivative asset types
* Improves comments on `StandardDeviationOfReturns` for clarity on how
to use the volatility model for end users
* Fixes bug where TradeBar could not have proper Symbol set when getting
max resolution
* Applies to QCAlgorithm.Universe and StandardDeviationOfReturnsVolatilityModel
* Adds tests to check volatility model is updated at specified config intervals
* Address review: add shared method for (Relative)StandardDeviation
volatility models
* Adjusts logic to determine bar type
* Address review: order by TickType when getting configs inside volatility models
* Allow USDC in cashbook without USDC-USD pair
* Cover more stablecoin cases unique to our crypto brokers
* Add unit test
* Cleanup and expand test cases
* Add USDCEUR and USDCGBP to GDAX Symbols
* Add missing tickers to SPDB
* Cleanup test
- Testing net5 uncovered these algorithms to be undeterministic
- Adjusting AllShortableSymbolsCoarseSelectionRegressionAlgorithm
internal implementation
- Order removal of universe members will be deterministic, when the
entire universe is removed.
* Fixes Double to Decimal Cast in GetAnnualPerformance
`GetAnnualPerformance` raises an exception if the `AnnualPerformance` calculation returns a double that cannot be cast to decimal (smaller than `decimal.MinValue` or bigger than `decimal.MaxValue`).
See `ProbabilisticSharpeRatio` where the same solution was applied.
* Updates SPY Market Data
SPY is a key asset since it is the default benchmark, and any change can lead to different `Alpha` and `Beta`
* Updates Unit Tests to Reflect Data Update
* Updates Regression Tests to Reflect Data Update I
Most of the regression tests change because of updated data (market and factors) of SPY (default benchmark) while the total trade remain the same.
* Updates Regression Tests to Reflect Data Update II
The following regression tests were changed to adapt to adjusted prices and keep the total trades:
- `BacktestingBrokerageRegressionAlgorithm`
- `LimitIfTouchedRegressionAlgorithm`
- `PortfolioRebalanceOnCustomFuncRegressionAlgorithm`
- `SetAccountCurrencySecurityMarginModelRegressionAlgorithm`
- `StopLossOnOrderEventRegressionAlgorithm`
- `TimeInForceAlgorithm`
The following regression tests have more trades since adjusted prices allowed more 1-2 shares trades that were rounded down to zero before:
- `FreePortfolioValueRegressionAlgorithm` 2 -> 3
- `PortfolioRebalanceOnDateRulesRegressionAlgorithm` 291 -> 298
- `TrailingStopRiskFrameworkAlgorithm` 5 -> 7
Especial cases:
- `AutoRegressiveIntegratedMovingAverageRegressionAlgorithm` 65 -> 52
- ARIMA model sensibility
- `BlackLittermanPortfolioOptimizationFrameworkAlgorithm` 18 -> 19
- BLM model sensibility
- `ExtendedMarketHoursHistoryRegressionAlgorithm` 20 -> 18
- Less minute bars before market opens
* Addresses Peer-Review
Fix `BacktestingBrokerageRegressionAlgorithm` to use `CalculateOrderQuantity` and round down `quantity` to an even number to pass a value assertion and update the expected value from 50 to 52.
The quantity calculated by `CalculateOrderQuantity` has changed from 50 to 53 because of factor file update.
* Add futures regression reproducing the issue
* Cleanup futures regression
* Add Options regression
* Add IRegressionAlgorithmDefinition
* Add IRegressionAlgorithmDefinition
* Adjust regressions to compare to expiration date; delisting time is not always correct
* Refactor subscription enumerator filtering of AuxData to top of the stack
* Update broken tests to reflect the split/dividends/delisting subscriptions
* Update regression statistics because of Delisting EOD instead of at time
* Let custom data configurations bypass filter
* Adjust expected count, now that we are letting aux data through to history requests
* another small adjustment
* Use ExpectedBarCount in error message
* Add filter test for both cases
* Add some clarifying comments
* Verify we did recieve data in the regressions
* Make _shouldEmitAuxiliaryData private readonly
* Filter out aux data for history requests
* Remove option to not include aux data in subscriptions
* Refactor filtering to be more explicit for each piece of data; fixes universe selection aux data
* Cleanup comments after removed var
* Refactor order of filtering for performance reasons.
`GetAnnualPerformance` raises an exception if the `AnnualPerformance` calculation returns a double that cannot be cast to decimal (smaller than `decimal.MinValue` or bigger than `decimal.MaxValue`).
See `ProbabilisticSharpeRatio` where the same solution was applied.
* Add SI
* Add ASI
* Add Swing Index
* Add Accumulative Swing Index
* Fix XML comment
* Add SI and ASI
* Add test data
* Add SI tests
* Add ASI tests
* Convert get only properties to methods
* Fix indicator name
* Replace special characters
* Fix indicator formula
* Replace test data
* Replace test data
* Update QCAlgorithm.Indicators.cs
* Minor format tweaks
Co-authored-by: Jared <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
- Fixed GDAXBrokerage missing fills caused by an incorrect assumption of a global counter for order fills (it is actually per-symbol)
- Filtered out rate limit messages in the fill monitor task
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonVirtualEnvironmentTests.AssertVirtualEnvironment"
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonVirtualEnvironmentTests.AssertVirtualEnvironment"
# Run Python Package Tests
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests" --blame-hang-timeout 120seconds --blame-crash
# Run StableBaselines Python Package Test
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.StableBaselinesTest" --blame-hang-timeout 120seconds --blame-crash
# Run AxPlatform Python Package Test
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.AxPlatformTest" --blame-hang-timeout 120seconds --blame-crash
# Run TensorlyTest Python Package Test
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.TensorlyTest" --blame-hang-timeout 120seconds --blame-crash
# Run NeuralTangents, Ignite Python Package Test
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.NeuralTangentsTest|IgniteTest" --blame-hang-timeout 120seconds --blame-crash
# Run TensorflowTest
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.TensorflowTest" --blame-hang-timeout 120seconds --blame-crash
# Run TensorflowProbability
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.TensorflowProbabilityTest" --blame-hang-timeout 120seconds --blame-crash
# Run Hvplot Python Package Test
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.HvplotTest" --blame-hang-timeout 120seconds --blame-crash
# Run Keras Python Package Test
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.KerasTest" --blame-hang-timeout 120seconds --blame-crash
# Run Transformers
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.Transformers" --blame-hang-timeout 120seconds --blame-crash
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.XTransformers" --blame-hang-timeout 120seconds --blame-crash
# Run Shap
dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "FullyQualifiedName=QuantConnect.Tests.Python.PythonPackagesTests.ShapTest" --blame-hang-timeout 120seconds --blame-crash
<h1>Develop Algorithms Locally, Run in Container</h1>
We have set up a relatively easy way to develop algorithms in your local IDE and push them into the container to be run and debugged.
Before we can use this method with Windows or Mac OS we need to share the Lean directory with Docker.
<br />
<h2>Activate File Sharing for Docker:</h2>
* Windows:
* [Guide to sharing](https://docs.docker.com/docker-for-windows/#file-sharing)
* Share the LEAN root directory with docker
* Mac:
* [Guide to sharing](https://docs.docker.com/docker-for-mac/#file-sharing)
* Share the LEAN root directory with docker
* Linux:
* (No setup required)
<br />
<h2>Lean Configuration</h2>
Next we need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set. Just like running lean locally the config must reflect what we want Lean to run.
You configuration file should look something like this:
Our specific configuration binds the Algorithm.Python directory to the container by default so any algorithm you would like to run should be in that directory. Please ensure your algorithm location looks just the same as the example above. If you want to use a different location refer to the section bellow on setting that argument for the container and make sure your config.json also reflects this.
<br />
<h2>Running Lean in the Container</h2>
This section will cover how to actually launch Lean in the container with your desired configuration.
From a terminal; Pycharm has a built in terminal on the bottom taskbar labeled **Terminal**; launch the run_docker.bat/.sh script; there are a few choices on how to launch this:
1. Launch with no parameters and answer the questions regarding configuration (Press enter for defaults)
* Would you like to debug C#? (Requires mono debugger attachment) [default: N]:
2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
IMAGE=quantconnect/lean:latest
CONFIG_FILE=
DATA_DIR=
RESULTS_DIR=
DEBUGGING=
PYTHON_DIR=
3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat DEBUGGING=y_**
* Accepted args for inline include all listed in the file in #2; must follow the **key=value** format
<br />
<h1>Debugging Python</h1>
Debugging your Python algorithms requires an extra step within your configuration and inside of PyCharm. Thankfully we were able to configure the PyCharm launch configurations to take care of most of the work for you!
<br />
<h2>Modifying the Configuration</h2>
First in order to debug a Python algorithm in Pycharm we must make the following change to our configuration (Launcher\config.json) under the comment debugging configuration:
"debugging": true,
"debugging-method": "PyCharm",
In setting this we are telling Lean to reach out and create a debugger connection using PyCharm’s PyDevd debugger server. Once this is set Lean will **always** attempt to connect to a debugger server on launch. **If you are no longer debugging set “debugging” to false.**
<br />
<h2>Using PyCharm Launch Options</h2>
Now that Lean is configured for the debugger we can make use of the programmed launch options to connect.
**<h3>Container (Recommended)</h3>**
To debug inside of the container we must first start the debugger server in Pycharm, to do this use the drop down configuration “Debug in Container” and launch the debugger. Be sure to set some breakpoints in your algorithms!
Then we will need to launch the container, follow the steps described in the section “[Running Lean in the Container](#Running-Lean-in-the-Container)”. After launching the container the debugging configuration will take effect and it will connect to the debug server where you can begin debugging your algorithm.
**<h3>Local</h3>**
To debug locally we must run the program locally. First, just as the container setup, start the PyCharm debugger server by running the “Debug Local” configuration.
Then start the program locally by whatever means you typically use, such as Mono, directly running the program at **QuantConnect.Lean.Launcher.exe**, etc. Once the program is running it will make the connection to your PyCharm debugger server where you can begin debugging your algorithm.
<h1>Local Development & Docker Integration with Visual Studio</h1>
<h1>Local Development with Visual Studio</h1>
This document contains information regarding ways to use Visual Studio to work with the Lean engine, this includes a couple options that make lean easy to develop on any machine:
This document contains information regarding ways to use Visual Studio to work with the Lean's Docker image.
- Using Lean CLI -> A great tool for working with your algorithms locally, while still being able to deploy to the cloud and have access to Lean data. It is also able to run algorithms locally through our official docker images **Recommended for algorithm development.
- Locally installing all dependencies to run Lean with Visual Studio on your OS.
<br />
<h1>Getting Setup</h1>
<h1>Setup</h1>
<h2>Option 1: Lean CLI</h2>
Before anything we need to ensure a few things have been done:
1. Get [Visual Studio](https://code.visualstudio.com/download)
* Get the Extension [VSMonoDebugger](https://marketplace.visualstudio.com/items?itemName=GordianDotNet.VSMonoDebugger0d62) for C# Debugging
2. Get [Docker](https://docs.docker.com/get-docker/):
* Follow the instructions for your Operating System
* New to Docker? Try docker getting-started
3. Pull Lean’s latest image from a terminal
*_docker pull quantconnect/lean_
4. Get Lean into Visual Studio
* Download the repo or clone it using: _git clone[ https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)_
* Open the solution **QuantConnect.Lean.sln** using Visual Studio
To use Lean CLI follow the instructions for installation and tutorial for usage in our [documentation](https://www.quantconnect.com/docs/v2/lean-cli/getting-started/lean-cli).
<br />
<h1>Develop Algorithms Locally, Run in Container</h1>
<h2>Option 2: Install Locally</h2>
1. Install [.Net 6](https://dotnet.microsoft.com/download) for the project
We have set up a relatively easy way to develop algorithms in your local IDE and push them into the container to be run and debugged.
2. (Optional) Get [Python 3.8.13](https://www.python.org/downloads/release/python-3813/) for running Python algorithms
- Follow Python instructions [here](https://github.com/QuantConnect/Lean/tree/master/Algorithm.Python#installing-python-38) for your platform
Before we can use this method with Windows or Mac OS we need to share the Lean directory with Docker.
3. Get [Visual Studio](https://visualstudio.microsoft.com/vs/)
4. Get Lean into VS
- Download the repo or clone it using: _git clone [https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)_
- Open the project file with VS (QuantConnect.Lean.sln)
Your environment is prepared and ready to run lean
<br />
<h2>Activate File Sharing for Docker:</h2>
<h1>How to use Lean</h1>
* Windows:
* [Guide to sharing](https://docs.docker.com/docker-for-windows/#file-sharing)
* Share the LEAN root directory with docker
* Mac:
* [Guide to sharing](https://docs.docker.com/docker-for-mac/#file-sharing)
* Share the LEAN root directory with docker
* Linux:
* (No setup required)
This section will cover configuring, launching and debugging lean. This is only applicable to option 2 from above. This does not apply to Lean CLI, please refer to [CLI documentation](https://www.quantconnect.com/docs/v2/lean-cli/getting-started/lean-cli)
<br />
<h2>Lean Configuration</h2>
<h2>Configuration</h2>
Next we need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set. Just like running lean locally the config must reflect what we want Lean to run.
We need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set.
You configuration file should look something like this for the following languages:
Your configuration file should look something like this for the following languages:
<h3>Python:</h3>
@@ -78,59 +63,11 @@ You configuration file should look something like this for the following languag
<br />
<h2>Important Note About C#</h2>
<h2>Launching Lean</h2>
In order to use a custom C# algorithm, the C# file must be compiled before running in the docker, as it is compiled into the file **"QuantConnect.Algorithm.CSharp.dll"**. Any new C# files will need to be added to the csproj compile list before it will compile, check **Algorithm.CSharp/QuantConnect.Algorithm.CSharp.csproj** for all algorithms that are compiled. Once there is an entry for your algorithm the project can be compiled by using **Build > Build Solution**.
If you would like to debug this file in the docker container one small change to the solutions target build is required.
1. Right click on the solution **QuantConnect.Lean** in the _Solution Explorer_
2. Select **Properties**
3. For project entry **QuantConnect.Algorithm.CSharp** change the configuration to **DebugDocker**
4. Select **Apply** and close out of the window.
5. Build the project at least once before running the docker.
Now that lean is configured we can launch. Use Visual Studio's run option, Make sure QuantConnect.Lean.Launcher is selected as the launch project. Any breakpoints in Lean C# will be triggered.
<br />
<h2>Running Lean in the Container</h2>
This section will cover how to actually launch Lean in the container with your desired configuration.
From a terminal launch the run_docker.bat/.sh script; there are a few choices on how to launch this:
1. Launch with no parameters and answer the questions regarding configuration (Press enter for defaults)
* Would you like to debug C#? (Requires mono debugger attachment) [default: N]:
2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
IMAGE=quantconnect/lean:latest
CONFIG_FILE=
DATA_DIR=
RESULTS_DIR=
DEBUGGING=
PYTHON_DIR=
3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat DEBUGGING=y_**
* Accepted args for inline include all listed in the file in #2
<br />
<h1>Connecting to Mono Debugger</h1>
If you launch the script with debugging set to **yes** (y), then you will need to connect to the debugging server with the mono extension that you installed in the setup stage.
To setup the extension do the following:
* Go to **Extensions > Mono > Settings...**
* Enter the following for the settings:
* Remote Host IP: 127.0.0.1
* Remote Host Port: 55555
* Mono Debug Port: 55555
* Click **Save** and then close the extension settings
Now that the extension is setup use it to connect to the Docker container by using:
***Extensions > Mono > Attach to mono debugger**
The program should then launch and trigger any breakpoints you have set in your C# Algorithm.
<h1>Common Issues</h1>
Here we will cover some common issues with setting this up. Feel free to contribute to this section!
<h1>Local Development & Docker Integration with Visual Studio Code</h1>
This document contains information regarding ways to use Visual Studio Code to work with the Lean engine, this includes a couple options that make lean easy to develop on any machine:
This document contains information regarding ways to use Visual Studio Code to work with the Lean engine, this includes using Lean’s Docker image in conjunction with local development as well as running Lean locally.
- Using Lean CLI -> A great tool for working with your algorithms locally, while still being able to deploy to the cloud and have access to Lean data. It is also able to run algorithms locally through our official docker images **Recommended for algorithm development.
- Using a Lean Dev container -> A docker environment with all dependencies pre-installed to allow seamless Lean development across platforms. Great for open source contributors.
- Locally installing all dependencies to run Lean with Visual Studio Code on your OS.
<br />
<h1>Getting Setup</h1>
<h1>Setup</h1>
<h2>Option 1: Lean CLI</h2>
Before anything we need to ensure a few things have been done:
To use Lean CLI follow the instructions for installation and tutorial for usage in our [documentation](https://www.quantconnect.com/docs/v2/lean-cli/key-concepts/getting-started)
<br />
<h2>Option 2: Lean Development Container</h2>
Before anything we need to ensure a few things have been done for either option:
1. Get [Visual Studio Code](https://code.visualstudio.com/download)
* Get the Extension [Mono Debug **15.8**](https://marketplace.visualstudio.com/items?itemName=ms-vscode.mono-debug) for C# Debugging
* Get the Extension [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for Python Debugging
- Get [Remote Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) Extension
2. Get [Docker](https://docs.docker.com/get-docker/):
* Follow the instructions for your Operating System
* New to Docker? Try docker getting-started
- Follow the instructions for your Operating System
- New to Docker? Try [docker getting-started](https://docs.docker.com/get-started/)
3.Install a compiler for the project **(Only needed for C# Debugging or Running Locally)**
* Visual Studio comes packed with msbuild or download without VS [here](https://visualstudio.microsoft.com/downloads/?q=build+tools)
* Put msbuild on your system path and test with command: _msbuild -version_
3.Pull Lean’s latest research image from a terminal
-`docker pull quantconnect/research:latest`
4.Pull Lean’s latest image from a terminal
*_docker pull quantconnect/lean_
4.Get Lean into VS Code
-Download the repo or clone it using: `git clone [https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)`
- Open the folder using VS Code
5.Get Lean into VS Code
* Download the repo or clone it using: _git clone[ https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)_
* Open the folder using VS Code
5.Open Development Container
-In VS Code, either:
-Select "Reopen in Container" from pop up box.
**NOTES**:
- Mono Extension Version 16 and greater fails to debug the docker container remotely, please install **Version 15.8**. To install an older version from within VS Code go to the extensions tab, search "Mono Debug", and select "Install Another Version...".
<br />
OR
<h1>Develop Algorithms Locally, Run in Container</h1>
- Ctrl+Shift+P (Command Palette) and select "Remote-Containers: Rebuild and Reopen in Container"
We have set up a relatively easy way to develop algorithms in your local IDE and push them into the container to be run and debugged.
Before we can use this method with Windows or Mac OS we need to share the Lean directory with Docker.
You should now be in the development container, give VS Code a moment to prepare and you will be ready to go!
If you would like to mount any additional local files to your container, checkout [devcontainer.json "mounts" section](https://containers.dev/implementors/json_reference/) for an example! Upon any mount changes you must rebuild the container using Command Palette as in step 5.
<br />
<h2>Activate File Sharing for Docker:</h2>
<h2>Option 3: Install Dependencies Locally</h2>
* Windows:
* [Guide to sharing](https://docs.docker.com/docker-for-windows/#file-sharing)
* Share the LEAN root directory with docker
* Mac:
* [Guide to sharing](https://docs.docker.com/docker-for-mac/#file-sharing)
* Share the LEAN root directory with docker
1. Install [.NET 6](https://dotnet.microsoft.com/en-us/download/dotnet/6.0) for the project
* Linux:
* (No setup required)
2. (Optional) Get [Python 3.8.13](https://www.python.org/downloads/release/python-3813/) for running Python algorithms
-Follow Python instructions [here](https://github.com/QuantConnect/Lean/tree/master/Algorithm.Python#installing-python-38) for your platform
3. Get [Visual Studio Code](https://code.visualstudio.com/download)
- Get the Extension [C#](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) for C# Debugging
- Get the Extension [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for Python Debugging
4. Get Lean into VS Code
- Download the repo or clone it using: `git clone [https://github.com/QuantConnect/Lean](https://github.com/QuantConnect/Lean)`
- Open the folder using VS Code
Your environment is prepared and ready to run Lean.
<br />
<h2>Lean Configuration</h2>
<h1>How to use Lean</h1>
Next we need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set. Just like running lean locally the config must reflect what we want Lean to run.
This section will cover configuring, building, launching and debugging lean. This is only applicable to option 2 from above. This does not apply to Lean CLI, please refer to [CLI documentation](https://www.quantconnect.com/docs/v2/lean-cli/key-concepts/getting-started)
<br />
<h2>Configuration</h2>
We need to be sure that our Lean configuration at **.\Launcher\config.json** is properly set.
Your configuration file should look something like this for the following languages:
@@ -85,105 +96,48 @@ Your configuration file should look something like this for the following langua
In order to use a custom C# algorithm, the C# file must be compiled before running in the docker, as it is compiled into the file "QuantConnect.Algorithm.CSharp.dll". Any new C# files will need to be added to the csproj compile list before it will compile, check Algorithm.CSharp/QuantConnect.Algorithm.CSharp.csproj for all algorithms that are compiled. Once there is an entry for your algorithm the project can be compiled by using the “build” task under _“Terminal” > “Run Build Task”._
Before running Lean, we must build the project. Currently the VS Code task will automatically build before launching. But find more information below about how to trigger building manually.
Python **does not** have this requirement as the engine will compile it on the fly.
In VS Code run build task (Ctrl+Shift+B or "Terminal" dropdown); there are a few options:
- __Build__ - basic build task, just builds Lean once
- __Rebuild__ - rebuild task, completely rebuilds the project. Use if having issues with debugging symbols being loaded for your algorithms.
- __Clean__ - deletes out all project build files
<br />
<h2>Running Lean in the Container</h2>
<h2>Launching Lean</h2>
This section will cover how to actually launch Lean in the container with your desired configuration.
Now that lean is configured and built we can launch Lean. Under "Run & Debug" use the launch option "Launch". This will start Lean with C# debugging. Any breakpoints in Lean C# will be triggered.
<br />
<h3>Option 1 (Recommended)</h3>
In VS Code click on the debug/run icon on the left toolbar, at the top you should see a drop down menu with launch options, be sure to select **Debug in Container**. This option will kick off a launch script that will start the docker. With this specific launch option the parameters are already configured in VS Codes **tasks.json** under the **run-docker** task args. These set arguments are:
As defaults these are all great! Feel free to change them as needed for your setup.
**NOTE:** VSCode may try and throw errors when launching this way regarding build on `QuantConnect.csx` and `Config.json` these errors can be ignored by selecting "*Debug Anyway*". To stop this error message in the future select "*Remember my choice in user settings*".
If using C# algorithms ensure that msbuild can build them successfully.
<h2>Debugging Python</h2>
Python algorithms require a little extra work in order to be able to debug them. Follow the steps below to get Python debugging working.
<br />
<h3>Option 2</h3>
From a terminal launch the run_docker.bat/.sh script; there are a few choices on how to launch this:
1. Launch with no parameters and answer the questions regarding configuration (Press enter for defaults)
* Would you like to debug C#? (Requires mono debugger attachment) [default: N]:
2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
IMAGE=quantconnect/lean:latest
CONFIG_FILE=
DATA_DIR=
RESULTS_DIR=
DEBUGGING=
PYTHON_DIR=
3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat DEBUGGING=y_**
* Accepted args for inline include all listed in the file in #2
<br />
<h1>Debugging Python</h1>
Python algorithms require a little extra work in order to be able to debug them locally or in the container. Thankfully we were able to configure VS code tasks to take care of the work for you! Follow the steps below to get Python debugging working.
<br />
<h2>Modifying the Configuration</h2>
<h3>Modifying the Configuration</h3>
First in order to debug a Python algorithm in VS Code we must make the following change to our configuration (Launcher\config.json) under the comment debugging configuration:
"debugging": true,
"debugging-method": "PTVSD",
"debugging-method": "DebugPy",
In setting this we are telling Lean to expect a debugger connection using ‘Python Tools for Visual Studio Debugger’. Once this is set Lean will stop upon initialization and await a connection to the debugger via port 5678.
<br />
<h2>Using VS Code Launch Options to Connect</h2>
<h3>Using VS Code Launch Options to Connect</h3>
Now that Lean is configured for the python debugger we can make use of the programmed launch options to connect.
Now that Lean is configured for the python debugger we can make use of the programmed launch options to connect to Lean during runtime.
<br />
<h3>Container</h3>
To debug inside of the container we must first start the container, follow the steps described in the section “[Running Lean in the Container](#Running-Lean-in-the-Container)”. Once the container is started you should see the messages in Figure 2.
If the message is displayed, use the same drop down for “Debug in Container” and select “Attach to Python (Container)”. Then press run, VS Code will now enter and debug any breakpoints you have set in your Python algorithm.
<br />
<h3>Local</h3>
To debug locally we must run the program locally using the programmed task found under Terminal > Run Task > “Run Application”. Once Lean is started you should see the messages in Figure 2.
If the message is displayed, use the launch option “Attach to Python (Local)”. Then press run, VS Code will now enter and debug any breakpoints you have set in your python algorithm.
Start Lean using the "Launch" option covered above. Once Lean starts you should see the messages in figure 2 If the message is displayed, use the launch option “Attach to Python”. Then press run, VS Code will now enter and debug any breakpoints you have set in your python algorithm.
Here we will cover some common issues with setting this up. This section will expand as we get user feedback!
* Any error messages about building in VSCode that point to comments in JSON. Either select **ignore** or follow steps described [here](https://stackoverflow.com/questions/47834825/in-vs-code-disable-error-comments-are-not-permitted-in-json) to remove the errors entirely.
*`Errors exist after running preLaunchTask 'run-docker'`This VSCode error appears to warn you of CSharp errors when trying to use `Debug in Container` select "Debug Anyway" as the errors are false flags for JSON comments as well as `QuantConnect.csx` not finding references. Neither of these will impact your debugging.
*`The container name "/LeanEngine" is already in use by container "****"` This Docker error implies that another instance of lean is already running under the container name /LeanEngine. If this error appears either use Docker Desktop to delete the container or use `docker kill LeanEngine` from the command line.
-The "project file cannot be loaded" and "nuget packages not found" errors occurs when the project files are open by another process in the host. Closing all applications and/or restarting the computer solve the issue.
-Autocomplete and reference finding with omnisharp can sometimes be buggy, if this occurs use the command palette to restart omnisharp. (Ctrl+Shift+P "OmniSharp: Restart OmniSharp")
-Any error messages about building in VSCode that point to comments in JSON. Either select **ignore** or follow steps described [here](https://stackoverflow.com/questions/47834825/in-vs-code-disable-error-comments-are-not-permitted-in-json) to remove the errors entirely.
thrownewException($"{Time}:: Detected differences in expected and actual securities{Environment.NewLine}Expected:{Environment.NewLine}{expected}{Environment.NewLine}Actual:{Environment.NewLine}{actual}");
thrownewException($"{Time}:: Detected differences in expected and actual universes{Environment.NewLine}Expected:{Environment.NewLine}{expected}{Environment.NewLine}Actual:{Environment.NewLine}{actual}");
thrownewException($"This algorithm intends to add a single security at a time but added: {changes.AddedSecurities.Count}{Environment.NewLine}{securities}");
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the evemts</param>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
/// Order fill event handler. On an order fill update the resulting information is passed to this method.
/// </summary>
/// <param name="orderEvent">Order event details containing details of the evemts</param>
/// <param name="orderEvent">Order event details containing details of the events</param>
/// <remarks>This method can be called asynchronously and so should only be used by seasoned C# experts. Ensure you use proper locks on thread-unsafe objects</remarks>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.