Quiver Quantitative Integration (#4869)

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverHouseDataDownloader.cs

* Delete QuiverSenateDataDownloader.cs

* Delete QuiverPoliticalBetaDataDownloader.cs

* Add files via upload

* Delete QuiverHouse.cs

* Delete QuiverSenate.cs

* Delete QuiverPoliticalBeta.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverDataAlgorithm.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Addresses self review: Cleans up code and adds new unit tests

  * Adds Quiver* C# files to project
  * Adds new unit test for QuiverCongress
  * Adds Python algorithm example

* Address self reviews

- Adding some missing xml docs
- Removing unrequired imports.
- Minor rename from Date to ReportDate
- Live trading will throw InvalidOperationException

* Fixes for example algorithms

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>

.gitignore

@@ -271,3 +271,6 @@ QuantConnect.Lean.sln.DotSettings*
 
 #User notebook files
 Research/Notebooks
+
+#Docker result files
+Results/

.idea/readme.md

@@ -91,14 +91,14 @@ From a terminal; Pycharm has a built in terminal on the bottom taskbar labeled *
 
  2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
   
-        image=quantconnect/lean:latest
-        config_file=
-        data_dir=
-        results_dir=
-        debugging=
-        python_dir=
+        IMAGE=quantconnect/lean:latest
+        CONFIG_FILE=
+        DATA_DIR=
+        RESULTS_DIR=
+        DEBUGGING=
+        PYTHON_DIR=
 
- 3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat debugging=y_** 
+ 3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat DEBUGGING=y_** 
       *    Accepted args for inline include all listed in the file in #2; must follow the **key=value** format
 
 <br />

.travis.yml

@@ -23,4 +23,4 @@ script:
   - msbuild /p:Configuration=Release /p:VbcToolExe=vbnc.exe QuantConnect.Lean.sln
   - mono ./testrunner/NUnit.ConsoleRunner.3.11.1/tools/nunit3-console.exe ./Tests/bin/Release/QuantConnect.Tests.dll --where "cat != TravisExclude" --labels=Off
   - chmod +x ci_build_stubs.sh
-  - sudo -E ./ci_build_stubs.sh -ipy -g -p
+  - sudo -E ./ci_build_stubs.sh -d -t -g -p

.vs/readme.md

@@ -106,14 +106,14 @@ From a terminal launch the run_docker.bat/.sh script; there are a few choices on
 
  2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
   
-        image=quantconnect/lean:latest
-        config_file=
-        data_dir=
-        results_dir=
-        debugging=
-        python_dir=
+        IMAGE=quantconnect/lean:latest
+        CONFIG_FILE=
+        DATA_DIR=
+        RESULTS_DIR=
+        DEBUGGING=
+        PYTHON_DIR=
 
- 3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat debugging=y_** 
+ 3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat DEBUGGING=y_** 
       *    Accepted args for inline include all listed in the file in #2
 
 <br />

.vscode/readme.md

@@ -101,14 +101,14 @@ This section will cover how to actually launch Lean in the container with your d
 
 <h3>Option 1 (Recommended)</h3>
 
-In VS Code click on the debug/run icon on the left toolbar, at the top you should see a drop down menu with launch options, be sure to select **Debug in Container**. This option will kick off a launch script that will start the docker. With this specific launch option the parameters are already configured in VS Codes **tasks.json** under the **run-docker** task args. These set arguements are:
+In VS Code click on the debug/run icon on the left toolbar, at the top you should see a drop down menu with launch options, be sure to select **Debug in Container**. This option will kick off a launch script that will start the docker. With this specific launch option the parameters are already configured in VS Codes **tasks.json** under the **run-docker** task args. These set arguments are:
 
-    "image=quantconnect/lean:latest",
-    "config_file=${workspaceFolder}/Launcher/config.json",
-    "data_dir=${workspaceFolder}/Data",
-    "results_dir=${workspaceFolder}/",
-    "debugging=Y",
-    "python_location=${workspaceFolder}/Algorithm.Python"
+    "IMAGE=quantconnect/lean:latest",
+    "CONFIG_FILE=${workspaceFolder}/Launcher/config.json",
+    "DATA_DIR=${workspaceFolder}/Data",
+    "RESULTS_DIR=${workspaceFolder}/Results",
+    "DEBUGGING=Y",
+    "PYHTON_DIR=${workspaceFolder}/Algorithm.Python"
 
 As defaults these are all great! Feel free to change them as needed for your setup.
 
@@ -120,21 +120,21 @@ From a terminal launch the run_docker.bat/.sh script; there are a few choices on
  1. Launch with no parameters and answer the questions regarding configuration (Press enter for defaults)
    
         *   Enter docker image [default: quantconnect/lean:latest]:
-        *   Enter absolute path to Lean config file [default: _~currentDir_\Launcher\config.json]:
-        *   Enter absolute path to Data folder [default: ~_currentDir_\Data\]:
-        *   Enter absolute path to store results [default: ~_currentDir_\]:
+        *   Enter absolute path to Lean config file [default: .\Launcher\config.json]:
+        *   Enter absolute path to Data folder [default: .\Data\]:
+        *   Enter absolute path to store results [default: .\Results]:
         *   Would you like to debug C#? (Requires mono debugger attachment) [default: N]:
 
  2. Using the **run_docker.cfg** to store args for repeated use; any blank entries will resort to default values! example: **_./run_docker.bat run_docker.cfg_**
   
-        image=quantconnect/lean:latest
-        config_file=
-        data_dir=
-        results_dir=
-        debugging=
-        python_dir=
+        IMAGE=quantconnect/lean:latest
+        CONFIG_FILE=
+        DATA_DIR=
+        RESULTS_DIR=
+        DEBUGGING=
+        PYTHON_DIR=
 
- 3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat debugging=y_** 
+ 3. Inline arguments; anything you don't enter will use the default args! example: **_./run_docker.bat DEBUGGING=y_** 
       *    Accepted args for inline include all listed in the file in #2
 
 <br />

.vscode/tasks.json

@@ -51,13 +51,13 @@
                 "command": "${workspaceFolder}/run_docker.sh"
             },
             "args": [
-                "image=quantconnect/lean:latest",
-                "config_file=${workspaceFolder}/Launcher/config.json",
-                "data_dir=${workspaceFolder}/Data",
-                "results_dir=${workspaceFolder}/",
-                "debugging=Y",
-                "python_dir=${workspaceFolder}/Algorithm.Python",
-                "exit=Y"
+                "IMAGE=quantconnect/lean:latest",
+                "CONFIG_FILE=${workspaceFolder}/Launcher/config.json",
+                "DATA_DIR=${workspaceFolder}/Data",
+                "RESULTS_DIR=${workspaceFolder}/Results",
+                "DEBUGGING=Y",
+                "PYTHON_DIR=${workspaceFolder}/Algorithm.Python",
+                "EXIT=Y"
             ],
             "problemMatcher": [
                 {

Algorithm.CSharp/AltData/QuiverWallStreetBetsDataAlgorithm.cs

@@ -0,0 +1,62 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using System.Linq;
+using QuantConnect.Data;
+using QuantConnect.Data.Custom.Quiver;
+
+namespace QuantConnect.Algorithm.CSharp.AltData
+{
+    /// <summary>
+    /// Quiver Quantitative is a provider of alternative data.
+    /// This algorithm shows how to consume the <see cref="QuiverWallStreetBets"/>
+    /// </summary>
+    public class QuiverWallStreetBetsDataAlgorithm : QCAlgorithm
+    {
+        /// <summary>
+        /// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.
+        /// </summary>
+        public override void Initialize()
+        {
+            SetStartDate(2019, 1, 1);
+            SetEndDate(2020, 6, 1);
+            SetCash(100000);
+
+            var aapl = AddEquity("AAPL", Resolution.Daily).Symbol;
+            var quiverWSBSymbol = AddData<QuiverWallStreetBets>(aapl).Symbol;
+            var history = History<QuiverWallStreetBets>(quiverWSBSymbol, 60, Resolution.Daily);
+
+            Debug($"We got {history.Count()} items from our history request");
+        }
+
+        public override void OnData(Slice data)
+        {
+            var points = data.Get<QuiverWallStreetBets>();
+            foreach (var point in points.Values)
+            {
+                // Go long in the stock if it was mentioned more than 5 times in the WallStreetBets daily discussion
+                if (point.Mentions > 5)
+                {
+                    SetHoldings(point.Symbol.Underlying, 1);
+                }
+                // Go short in the stock if it was mentioned less than 5 times in the WallStreetBets daily discussion
+                if (point.Mentions < 5)
+                {
+                    SetHoldings(point.Symbol.Underlying, -1);
+                }
+            }
+        }
+    }
+}

Algorithm.CSharp/BasicTemplateAlgorithm.cs

@@ -119,4 +119,4 @@ namespace QuantConnect.Algorithm.CSharp
             {"OrderListHash", "491919591"}
         };
     }
-}
+}

Algorithm.CSharp/CustomBuyingPowerModelAlgorithm.cs

@@ -0,0 +1,138 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using System;
+using QuantConnect.Interfaces;
+using QuantConnect.Securities;
+using System.Collections.Generic;
+using QuantConnect.Data;
+
+namespace QuantConnect.Algorithm.CSharp
+{
+    /// <summary>
+    /// Demonstration of using custom buying power model in backtesting.
+    /// QuantConnect allows you to model all orders as deeply and accurately as you need.
+    /// </summary>
+    /// <meta name="tag" content="trading and orders" />
+    /// <meta name="tag" content="transaction fees and slippage" />
+    /// <meta name="tag" content="custom buying power models" />
+    public class CustomBuyingPowerModelAlgorithm : QCAlgorithm, IRegressionAlgorithmDefinition
+    {
+        private Symbol _spy;
+
+        public override void Initialize()
+        {
+            SetStartDate(2013, 10, 01);
+            SetEndDate(2013, 10, 31);
+            var security = AddEquity("SPY", Resolution.Hour);
+            _spy = security.Symbol;
+
+            // set the buying power model
+            security.SetBuyingPowerModel(new CustomBuyingPowerModel());
+        }
+
+        public void OnData(Slice slice)
+        {
+            if (Portfolio.Invested)
+            {
+                return;
+            }
+
+            var quantity = CalculateOrderQuantity(_spy, 1m);
+            if (quantity % 100 != 0)
+            {
+                throw new Exception($"CustomBuyingPowerModel only allow quantity that is multiple of 100 and {quantity} was found");
+            }
+
+            // We normally get insufficient buying power model, but the
+            // CustomBuyingPowerModel always says that there is sufficient buying power for the orders
+            MarketOrder(_spy, quantity * 10);
+        }
+
+        public class CustomBuyingPowerModel : BuyingPowerModel
+        {
+            public override GetMaximumOrderQuantityResult GetMaximumOrderQuantityForTargetBuyingPower(
+                GetMaximumOrderQuantityForTargetBuyingPowerParameters parameters)
+            {
+                var quantity = base.GetMaximumOrderQuantityForTargetBuyingPower(parameters).Quantity;
+                quantity = Math.Floor(quantity / 100) * 100;
+                return new GetMaximumOrderQuantityResult(quantity);
+            }
+
+            public override HasSufficientBuyingPowerForOrderResult HasSufficientBuyingPowerForOrder(
+                HasSufficientBuyingPowerForOrderParameters parameters)
+            {
+                return new HasSufficientBuyingPowerForOrderResult(true);
+            }
+        }
+
+        /// <summary>
+        /// This is used by the regression test system to indicate if the open source Lean repository has the required data to run this algorithm.
+        /// </summary>
+        public bool CanRunLocally { get; } = true;
+
+        /// <summary>
+        /// This is used by the regression test system to indicate which languages this algorithm is written in.
+        /// </summary>
+        public Language[] Languages { get; } = { Language.CSharp, Language.Python };
+
+        /// <summary>
+        /// This is used by the regression test system to indicate what the expected statistics are from running the algorithm
+        /// </summary>
+        public Dictionary<string, string> ExpectedStatistics => new Dictionary<string, string>
+        {
+            {"Total Trades", "1"},
+            {"Average Win", "0%"},
+            {"Average Loss", "0%"},
+            {"Compounding Annual Return", "5672.520%"},
+            {"Drawdown", "22.500%"},
+            {"Expectancy", "0"},
+            {"Net Profit", "40.601%"},
+            {"Sharpe Ratio", "40.201"},
+            {"Probabilistic Sharpe Ratio", "77.339%"},
+            {"Loss Rate", "0%"},
+            {"Win Rate", "0%"},
+            {"Profit-Loss Ratio", "0"},
+            {"Alpha", "41.848"},
+            {"Beta", "9.224"},
+            {"Annual Standard Deviation", "1.164"},
+            {"Annual Variance", "1.355"},
+            {"Information Ratio", "44.459"},
+            {"Tracking Error", "1.04"},
+            {"Treynor Ratio", "5.073"},
+            {"Total Fees", "$30.00"},
+            {"Fitness Score", "0.418"},
+            {"Kelly Criterion Estimate", "0"},
+            {"Kelly Criterion Probability Value", "0"},
+            {"Sortino Ratio", "113.05"},
+            {"Return Over Maximum Drawdown", "442.81"},
+            {"Portfolio Turnover", "0.418"},
+            {"Total Insights Generated", "0"},
+            {"Total Insights Closed", "0"},
+            {"Total Insights Analysis Completed", "0"},
+            {"Long Insight Count", "0"},
+            {"Short Insight Count", "0"},
+            {"Long/Short Ratio", "100%"},
+            {"Estimated Monthly Alpha Value", "$0"},
+            {"Total Accumulated Estimated Alpha Value", "$0"},
+            {"Mean Population Estimated Insight Value", "$0"},
+            {"Mean Population Direction", "0%"},
+            {"Mean Population Magnitude", "0%"},
+            {"Rolling Averaged Population Direction", "0%"},
+            {"Rolling Averaged Population Magnitude", "0%"},
+            {"OrderListHash", "639761089"}
+        };
+    }
+}

Algorithm.CSharp/CustomModelsAlgorithm.cs

@@ -26,11 +26,12 @@ using QuantConnect.Securities;
 namespace QuantConnect.Algorithm.CSharp
 {
     /// <summary>
-    /// Demonstration of using custom fee, slippage and fill models for modelling transactions in backtesting.
+    /// Demonstration of using custom fee, slippage, fill, and buying power models for modelling transactions in backtesting.
     /// QuantConnect allows you to model all orders as deeply and accurately as you need.
     /// </summary>
     /// <meta name="tag" content="trading and orders" />
     /// <meta name="tag" content="transaction fees and slippage" />
+    /// <meta name="tag" content="custom buying power models" />
     /// <meta name="tag" content="custom transaction models" />
     /// <meta name="tag" content="custom slippage models" />
     /// <meta name="tag" content="custom fee models" />
@@ -50,6 +51,7 @@ namespace QuantConnect.Algorithm.CSharp
             _security.SetFeeModel(new CustomFeeModel(this));
             _security.SetFillModel(new CustomFillModel(this));
             _security.SetSlippageModel(new CustomSlippageModel(this));
+            _security.SetBuyingPowerModel(new CustomBuyingPowerModel(this));
         }
 
         public void OnData(TradeBars data)
@@ -60,13 +62,13 @@ namespace QuantConnect.Algorithm.CSharp
             if (Time.Day > 10 && _security.Holdings.Quantity <= 0)
             {
                 var quantity = CalculateOrderQuantity(_spy, .5m);
-                Log("MarketOrder: " + quantity);
+                Log($"MarketOrder: {quantity}");
                 MarketOrder(_spy, quantity, asynchronous: true); // async needed for partial fill market orders
             }
             else if (Time.Day > 20 && _security.Holdings.Quantity >= 0)
             {
                 var quantity = CalculateOrderQuantity(_spy, -.5m);
-                Log("MarketOrder: " + quantity);
+                Log($"MarketOrder: {quantity}");
                 MarketOrder(_spy, quantity, asynchronous: true); // async needed for partial fill market orders
             }
         }
@@ -109,7 +111,7 @@ namespace QuantConnect.Algorithm.CSharp
                     fill.Status = OrderStatus.PartiallyFilled;
                 }
 
-                _algorithm.Log("CustomFillModel: " + fill);
+                _algorithm.Log($"CustomFillModel: {fill}");
 
                 return fill;
             }
@@ -131,7 +133,7 @@ namespace QuantConnect.Algorithm.CSharp
                     1m,
                     parameters.Security.Price*parameters.Order.AbsoluteQuantity*0.00001m);
 
-                _algorithm.Log("CustomFeeModel: " + fee);
+                _algorithm.Log($"CustomFeeModel: {fee}");
                 return new OrderFee(new CashAmount(fee, "USD"));
             }
         }
@@ -150,11 +152,31 @@ namespace QuantConnect.Algorithm.CSharp
                 // custom slippage math
                 var slippage = asset.Price*0.0001m*(decimal) Math.Log10(2*(double) order.AbsoluteQuantity);
 
-                _algorithm.Log("CustomSlippageModel: " + slippage);
+                _algorithm.Log($"CustomSlippageModel: {slippage}");
                 return slippage;
             }
         }
 
+        public class CustomBuyingPowerModel : BuyingPowerModel
+        {
+            private readonly QCAlgorithm _algorithm;
+
+            public CustomBuyingPowerModel(QCAlgorithm algorithm)
+            {
+                _algorithm = algorithm;
+            }
+
+            public override HasSufficientBuyingPowerForOrderResult HasSufficientBuyingPowerForOrder(
+                HasSufficientBuyingPowerForOrderParameters parameters)
+            {
+                // custom behavior: this model will assume that there is always enough buying power
+                var hasSufficientBuyingPowerForOrderResult = new HasSufficientBuyingPowerForOrderResult(true);
+                _algorithm.Log($"CustomBuyingPowerModel: {hasSufficientBuyingPowerForOrderResult.IsSufficient}");
+
+                return hasSufficientBuyingPowerForOrderResult;
+            }
+        }
+
         /// <summary>
         /// This is used by the regression test system to indicate if the open source Lean repository has the required data to run this algorithm.
         /// </summary>

Algorithm.CSharp/DropboxBaseDataUniverseSelectionAlgorithm.cs

@@ -187,12 +187,12 @@ namespace QuantConnect.Algorithm.CSharp
             {"Total Trades", "6441"},
             {"Average Win", "0.07%"},
             {"Average Loss", "-0.07%"},
-            {"Compounding Annual Return", "13.284%"},
+            {"Compounding Annual Return", "13.331%"},
             {"Drawdown", "10.700%"},
             {"Expectancy", "0.061"},
-            {"Net Profit", "13.284%"},
-            {"Sharpe Ratio", "0.96"},
-            {"Probabilistic Sharpe Ratio", "46.111%"},
+            {"Net Profit", "13.331%"},
+            {"Sharpe Ratio", "0.963"},
+            {"Probabilistic Sharpe Ratio", "46.232%"},
             {"Loss Rate", "46%"},
             {"Win Rate", "54%"},
             {"Profit-Loss Ratio", "0.97"},
@@ -200,15 +200,15 @@ namespace QuantConnect.Algorithm.CSharp
             {"Beta", "-0.066"},
             {"Annual Standard Deviation", "0.121"},
             {"Annual Variance", "0.015"},
-            {"Information Ratio", "0.004"},
+            {"Information Ratio", "0.006"},
             {"Tracking Error", "0.171"},
-            {"Treynor Ratio", "-1.754"},
-            {"Total Fees", "$8669.33"},
+            {"Treynor Ratio", "-1.761"},
+            {"Total Fees", "$8669.41"},
             {"Fitness Score", "0.675"},
             {"Kelly Criterion Estimate", "0"},
             {"Kelly Criterion Probability Value", "0"},
-            {"Sortino Ratio", "1.124"},
-            {"Return Over Maximum Drawdown", "1.242"},
+            {"Sortino Ratio", "1.127"},
+            {"Return Over Maximum Drawdown", "1.246"},
             {"Portfolio Turnover", "1.64"},
             {"Total Insights Generated", "0"},
             {"Total Insights Closed", "0"},
@@ -223,7 +223,7 @@ namespace QuantConnect.Algorithm.CSharp
             {"Mean Population Magnitude", "0%"},
             {"Rolling Averaged Population Direction", "0%"},
             {"Rolling Averaged Population Magnitude", "0%"},
-            {"OrderListHash", "-1120327913"}
+            {"OrderListHash", "-75671425"}
         };
     }
 }

Algorithm.CSharp/DropboxUniverseSelectionAlgorithm.cs

@@ -160,12 +160,12 @@ namespace QuantConnect.Algorithm.CSharp
             {"Total Trades", "5059"},
             {"Average Win", "0.08%"},
             {"Average Loss", "-0.08%"},
-            {"Compounding Annual Return", "14.901%"},
+            {"Compounding Annual Return", "14.950%"},
             {"Drawdown", "10.600%"},
             {"Expectancy", "0.075"},
-            {"Net Profit", "14.901%"},
-            {"Sharpe Ratio", "1.068"},
-            {"Probabilistic Sharpe Ratio", "50.201%"},
+            {"Net Profit", "14.950%"},
+            {"Sharpe Ratio", "1.072"},
+            {"Probabilistic Sharpe Ratio", "50.327%"},
             {"Loss Rate", "45%"},
             {"Win Rate", "55%"},
             {"Profit-Loss Ratio", "0.97"},
@@ -173,15 +173,15 @@ namespace QuantConnect.Algorithm.CSharp
             {"Beta", "-0.066"},
             {"Annual Standard Deviation", "0.121"},
             {"Annual Variance", "0.015"},
-            {"Information Ratio", "0.08"},
+            {"Information Ratio", "0.083"},
             {"Tracking Error", "0.171"},
-            {"Treynor Ratio", "-1.963"},
-            {"Total Fees", "$6806.57"},
+            {"Treynor Ratio", "-1.971"},
+            {"Total Fees", "$6806.67"},
             {"Fitness Score", "0.694"},
             {"Kelly Criterion Estimate", "0"},
             {"Kelly Criterion Probability Value", "0"},
-            {"Sortino Ratio", "1.261"},
-            {"Return Over Maximum Drawdown", "1.404"},
+            {"Sortino Ratio", "1.265"},
+            {"Return Over Maximum Drawdown", "1.409"},
             {"Portfolio Turnover", "1.296"},
             {"Total Insights Generated", "0"},
             {"Total Insights Closed", "0"},
@@ -196,7 +196,7 @@ namespace QuantConnect.Algorithm.CSharp
             {"Mean Population Magnitude", "0%"},
             {"Rolling Averaged Population Direction", "0%"},
             {"Rolling Averaged Population Magnitude", "0%"},
-            {"OrderListHash", "974523768"}
+            {"OrderListHash", "1142077166"}
         };
     }
 }

Algorithm.CSharp/HistoryWithSymbolChangesRegressionAlgorithm.cs

@@ -53,7 +53,7 @@ namespace QuantConnect.Algorithm.CSharp
             }
 
             var firstBar = history.First().Bars.GetValue(symbol);
-            if (firstBar.EndTime != new DateTime(1998, 3, 3) || firstBar.Close != 26.3607004m)
+            if (firstBar.EndTime != new DateTime(1998, 3, 3) || firstBar.Close != 25.11427695m)
             {
                 throw new Exception("First History bar - unexpected data received");
             }

Algorithm.CSharp/MaximumPortfolioDrawdownFrameworkAlgorithm.cs

@@ -65,32 +65,32 @@ namespace QuantConnect.Algorithm.CSharp
         /// </summary>
         public Dictionary<string, string> ExpectedStatistics => new Dictionary<string, string>
         {
-            {"Total Trades", "2"},
+            {"Total Trades", "5"},
             {"Average Win", "0%"},
-            {"Average Loss", "-0.98%"},
-            {"Compounding Annual Return", "-53.792%"},
-            {"Drawdown", "1.500%"},
+            {"Average Loss", "-0.52%"},
+            {"Compounding Annual Return", "246.602%"},
+            {"Drawdown", "2.300%"},
             {"Expectancy", "-1"},
-            {"Net Profit", "-0.982%"},
-            {"Sharpe Ratio", "-5.949"},
-            {"Probabilistic Sharpe Ratio", "1.216%"},
+            {"Net Profit", "1.602%"},
+            {"Sharpe Ratio", "8.065"},
+            {"Probabilistic Sharpe Ratio", "65.943%"},
             {"Loss Rate", "100%"},
             {"Win Rate", "0%"},
             {"Profit-Loss Ratio", "0"},
-            {"Alpha", "-0.973"},
-            {"Beta", "0.268"},
-            {"Annual Standard Deviation", "0.077"},
-            {"Annual Variance", "0.006"},
-            {"Information Ratio", "-14.167"},
-            {"Tracking Error", "0.168"},
-            {"Treynor Ratio", "-1.705"},
-            {"Total Fees", "$6.51"},
-            {"Fitness Score", "0.249"},
+            {"Alpha", "-0.157"},
+            {"Beta", "1.015"},
+            {"Annual Standard Deviation", "0.223"},
+            {"Annual Variance", "0.05"},
+            {"Information Ratio", "-27.079"},
+            {"Tracking Error", "0.005"},
+            {"Treynor Ratio", "1.772"},
+            {"Total Fees", "$16.28"},
+            {"Fitness Score", "0.999"},
             {"Kelly Criterion Estimate", "38.64"},
             {"Kelly Criterion Probability Value", "0.229"},
             {"Sortino Ratio", "79228162514264337593543950335"},
-            {"Return Over Maximum Drawdown", "-55.465"},
-            {"Portfolio Turnover", "0.498"},
+            {"Return Over Maximum Drawdown", "78.607"},
+            {"Portfolio Turnover", "1.246"},
             {"Total Insights Generated", "100"},
             {"Total Insights Closed", "99"},
             {"Total Insights Analysis Completed", "99"},
@@ -104,7 +104,7 @@ namespace QuantConnect.Algorithm.CSharp
             {"Mean Population Magnitude", "54.5455%"},
             {"Rolling Averaged Population Direction", "59.8056%"},
             {"Rolling Averaged Population Magnitude", "59.8056%"},
-            {"OrderListHash", "160051570"}
+            {"OrderListHash", "-1552239367"}
         };
     }
 }

Algorithm.CSharp/OptionRenameRegressionAlgorithm.cs

@@ -56,6 +56,13 @@ namespace QuantConnect.Algorithm.CSharp
         /// <param name="slice">The current slice of data keyed by symbol string</param>
         public override void OnData(Slice slice)
         {
+            foreach (var dividend in slice.Dividends.Values)
+            {
+                if (dividend.ReferencePrice != 32.59m || dividend.Distribution != 3.82m)
+                {
+                    throw new Exception($"{Time} - Invalid dividend {dividend}");
+                }
+            }
             if (!Portfolio.Invested)
             {
                 if (Time.Day == 28 && Time.Hour > 9 && Time.Minute > 0)
@@ -139,11 +146,11 @@ namespace QuantConnect.Algorithm.CSharp
             {"Total Trades", "4"},
             {"Average Win", "0%"},
             {"Average Loss", "-0.02%"},
-            {"Compounding Annual Return", "-0.453%"},
+            {"Compounding Annual Return", "-0.492%"},
             {"Drawdown", "0.000%"},
             {"Expectancy", "-1"},
             {"Net Profit", "-0.006%"},
-            {"Sharpe Ratio", "-3.554"},
+            {"Sharpe Ratio", "-3.943"},
             {"Probabilistic Sharpe Ratio", "0%"},
             {"Loss Rate", "100%"},
             {"Win Rate", "0%"},
@@ -152,15 +159,15 @@ namespace QuantConnect.Algorithm.CSharp
             {"Beta", "0"},
             {"Annual Standard Deviation", "0.002"},
             {"Annual Variance", "0"},
-            {"Information Ratio", "-3.554"},
+            {"Information Ratio", "-3.943"},
             {"Tracking Error", "0.002"},
             {"Treynor Ratio", "0"},
             {"Total Fees", "$4.00"},
-            {"Fitness Score", "0.001"},
+            {"Fitness Score", "0"},
             {"Kelly Criterion Estimate", "0"},
             {"Kelly Criterion Probability Value", "0"},
             {"Sortino Ratio", "79228162514264337593543950335"},
-            {"Return Over Maximum Drawdown", "-1.768"},
+            {"Return Over Maximum Drawdown", "-2.808"},
             {"Portfolio Turnover", "0.001"},
             {"Total Insights Generated", "0"},
             {"Total Insights Closed", "0"},

Algorithm.CSharp/QuantConnect.Algorithm.CSharp.csproj

@@ -142,7 +142,9 @@
       <Link>Properties\SharedAssemblyInfo.cs</Link>
     </Compile>
     <Compile Include="AddAlphaModelAlgorithm.cs" />
+    <Compile Include="CustomBuyingPowerModelAlgorithm.cs" />
     <Compile Include="AddOptionContractExpiresRegressionAlgorithm.cs" />
+    <Compile Include="AltData\QuiverWallStreetBetsDataAlgorithm.cs" />
     <Compile Include="ScaledFillForwardDataRegressionAlgorithm.cs" />
     <Compile Include="DailyHistoryForDailyResolutionRegressionAlgorithm.cs" />
     <Compile Include="DailyHistoryForMinuteResolutionRegressionAlgorithm.cs" />

Algorithm.CSharp/TotalPortfolioValueRegressionAlgorithm.cs

@@ -117,28 +117,28 @@ namespace QuantConnect.Algorithm.CSharp
             {"Total Trades", "3528"},
             {"Average Win", "0.67%"},
             {"Average Loss", "-0.71%"},
-            {"Compounding Annual Return", "17.318%"},
+            {"Compounding Annual Return", "17.227%"},
             {"Drawdown", "63.700%"},
             {"Expectancy", "0.020"},
-            {"Net Profit", "17.318%"},
-            {"Sharpe Ratio", "0.836"},
-            {"Probabilistic Sharpe Ratio", "33.715%"},
+            {"Net Profit", "17.227%"},
+            {"Sharpe Ratio", "0.834"},
+            {"Probabilistic Sharpe Ratio", "33.688%"},
             {"Loss Rate", "48%"},
             {"Win Rate", "52%"},
             {"Profit-Loss Ratio", "0.95"},
-            {"Alpha", "0.826"},
+            {"Alpha", "0.825"},
             {"Beta", "-0.34"},
             {"Annual Standard Deviation", "0.945"},
             {"Annual Variance", "0.893"},
-            {"Information Ratio", "0.714"},
+            {"Information Ratio", "0.713"},
             {"Tracking Error", "0.957"},
-            {"Treynor Ratio", "-2.325"},
-            {"Total Fees", "$24713.42"},
+            {"Treynor Ratio", "-2.323"},
+            {"Total Fees", "$24760.85"},
             {"Fitness Score", "0.54"},
             {"Kelly Criterion Estimate", "0"},
             {"Kelly Criterion Probability Value", "0"},
-            {"Sortino Ratio", "0.24"},
-            {"Return Over Maximum Drawdown", "0.272"},
+            {"Sortino Ratio", "0.238"},
+            {"Return Over Maximum Drawdown", "0.27"},
             {"Portfolio Turnover", "7.204"},
             {"Total Insights Generated", "0"},
             {"Total Insights Closed", "0"},
@@ -153,7 +153,7 @@ namespace QuantConnect.Algorithm.CSharp
             {"Mean Population Magnitude", "0%"},
             {"Rolling Averaged Population Direction", "0%"},
             {"Rolling Averaged Population Magnitude", "0%"},
-            {"OrderListHash", "-1547947497"}
+            {"OrderListHash", "843493486"}
         };
     }
 }

Algorithm.Framework/Risk/MaximumDrawdownPercentPortfolio.cs

@@ -67,9 +67,11 @@ namespace QuantConnect.Algorithm.Framework.Risk
             }
 
             var pnl = GetTotalDrawdownPercent(currentValue);
-            if (pnl < _maximumDrawdownPercent)
+            if (pnl < _maximumDrawdownPercent && targets.Length != 0)
             {
-                foreach(var target in targets)
+                // reset the trailing high value for restart investing on next rebalcing period
+                _initialised = false;
+                foreach (var target in targets)
                     yield return new PortfolioTarget(target.Symbol, 0);
             }
         }

Algorithm.Framework/Risk/MaximumDrawdownPercentPortfolio.py

@@ -54,10 +54,11 @@ class MaximumDrawdownPercentPortfolio(RiskManagementModel):
             return []   # return if new high reached
 
         pnl = self.GetTotalDrawdownPercent(currentValue)
-        if pnl < self.maximumDrawdownPercent:
+        if pnl < self.maximumDrawdownPercent and len(targets) != 0:
+            self.initialised = False # reset the trailing high value for restart investing on next rebalcing period
             return [ PortfolioTarget(target.Symbol, 0) for target in targets ]
 
         return []
 
     def GetTotalDrawdownPercent(self, currentValue):
-        return (float(currentValue) / float(self.portfolioHigh)) - 1.0
+        return (float(currentValue) / float(self.portfolioHigh)) - 1.0

Algorithm.Python/AltData/QuiverWallStreetBetsDataAlgorithm.py

@@ -0,0 +1,50 @@
+# QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+# Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from clr import AddReference
+AddReference("System")
+AddReference("QuantConnect.Algorithm")
+AddReference("QuantConnect.Common")
+
+from System import *
+from QuantConnect import *
+from QuantConnect.Algorithm import *
+from QuantConnect.Data import *
+from QuantConnect.Data.Custom.Quiver import *
+
+### <summary>
+### Quiver Quantitative is a provider of alternative data.
+### This algorithm shows how to consume the 'QuiverWallStreetBets'
+### </summary>
+class QuiverWallStreetBetsDataAlgorithm(QCAlgorithm):
+    def Initialize(self):
+        self.SetStartDate(2019, 1, 1)
+        self.SetEndDate(2020, 6, 1)
+        self.SetCash(100000)
+
+        aapl = self.AddEquity("AAPL", Resolution.Daily).Symbol
+        quiverWSBSymbol = self.AddData(QuiverWallStreetBets, aapl).Symbol
+        history = self.History(QuiverWallStreetBets, quiverWSBSymbol, 60, Resolution.Daily)
+
+        self.Debug(f"We got {len(history)} items from our history request");
+
+    def OnData(self, data):
+        points = data.Get(QuiverWallStreetBets)
+        for point in points.Values:
+            # Go long in the stock if it was mentioned more than 5 times in the WallStreetBets daily discussion
+            if point.Mentions > 5:
+                self.SetHoldings(point.Symbol.Underlying, 1)
+
+            # Go short in the stock if it was mentioned less than 5 times in the WallStreetBets daily discussion
+            if point.Mentions < 5:
+                self.SetHoldings(point.Symbol.Underlying, -1)

Algorithm.Python/CustomBuyingPowerModelAlgorithm.py

@@ -0,0 +1,65 @@
+# QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+# Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from clr import AddReference
+AddReference("System")
+AddReference("QuantConnect.Algorithm")
+AddReference("QuantConnect.Common")
+
+from System import *
+from QuantConnect import *
+from QuantConnect.Algorithm import *
+from QuantConnect.Securities import *
+import numpy as np
+
+### <summary>
+### Demonstration of using custom buying power model in backtesting.
+### QuantConnect allows you to model all orders as deeply and accurately as you need.
+### </summary>
+### <meta name="tag" content="trading and orders" />
+### <meta name="tag" content="transaction fees and slippage" />
+### <meta name="tag" content="custom buying power models" />
+class CustomBuyingPowerModelAlgorithm(QCAlgorithm):
+    '''Demonstration of using custom buying power model in backtesting.
+    QuantConnect allows you to model all orders as deeply and accurately as you need.'''
+
+    def Initialize(self):
+        self.SetStartDate(2013,10,1)   # Set Start Date
+        self.SetEndDate(2013,10,31)    # Set End Date
+        security = self.AddEquity("SPY", Resolution.Hour)
+        self.spy = security.Symbol
+
+        # set the buying power model
+        security.SetBuyingPowerModel(CustomBuyingPowerModel())
+
+    def OnData(self, slice):
+        if self.Portfolio.Invested:
+            return
+
+        quantity = self.CalculateOrderQuantity(self.spy, 1)
+        if quantity % 100 != 0:
+            raise Exception(f'CustomBuyingPowerModel only allow quantity that is multiple of 100 and {quantity} was found')
+
+        # We normally get insufficient buying power model, but the
+        # CustomBuyingPowerModel always says that there is sufficient buying power for the orders
+        self.MarketOrder(self.spy, quantity * 10)
+
+
+class CustomBuyingPowerModel(BuyingPowerModel):
+    def GetMaximumOrderQuantityForTargetBuyingPower(self, parameters):
+        quantity = super().GetMaximumOrderQuantityForTargetBuyingPower(parameters).Quantity
+        quantity = np.floor(quantity / 100) * 100
+        return GetMaximumOrderQuantityResult(quantity)
+
+    def HasSufficientBuyingPowerForOrder(self, parameters):
+        return HasSufficientBuyingPowerForOrderResult(True)

Algorithm.Python/CustomModelsAlgorithm.py

@@ -27,16 +27,17 @@ import numpy as np
 import random
 
 ### <summary>
-### Demonstration of using custom fee, slippage and fill models for modelling transactions in backtesting.
+### Demonstration of using custom fee, slippage, fill, and buying power models for modelling transactions in backtesting.
 ### QuantConnect allows you to model all orders as deeply and accurately as you need.
 ### </summary>
 ### <meta name="tag" content="trading and orders" />
 ### <meta name="tag" content="transaction fees and slippage" />
+### <meta name="tag" content="custom buying power models" />
 ### <meta name="tag" content="custom transaction models" />
 ### <meta name="tag" content="custom slippage models" />
 ### <meta name="tag" content="custom fee models" />
 class CustomModelsAlgorithm(QCAlgorithm):
-    '''Demonstration of using custom fee, slippage and fill models for modelling transactions in backtesting.
+    '''Demonstration of using custom fee, slippage, fill, and buying power models for modelling transactions in backtesting.
     QuantConnect allows you to model all orders as deeply and accurately as you need.'''
 
     def Initialize(self):
@@ -49,6 +50,7 @@ class CustomModelsAlgorithm(QCAlgorithm):
         self.security.SetFeeModel(CustomFeeModel(self))
         self.security.SetFillModel(CustomFillModel(self))
         self.security.SetSlippageModel(CustomSlippageModel(self))
+        self.security.SetBuyingPowerModel(CustomBuyingPowerModel(self))
 
 
     def OnData(self, data):
@@ -57,12 +59,12 @@ class CustomModelsAlgorithm(QCAlgorithm):
 
         if self.Time.day > 10 and self.security.Holdings.Quantity <= 0:
             quantity = self.CalculateOrderQuantity(self.spy, .5)
-            self.Log("MarketOrder: " + str(quantity))
+            self.Log(f"MarketOrder: {quantity}")
             self.MarketOrder(self.spy, quantity, True)   # async needed for partial fill market orders
 
         elif self.Time.day > 20 and self.security.Holdings.Quantity >= 0:
             quantity = self.CalculateOrderQuantity(self.spy, -.5)
-            self.Log("MarketOrder: " + str(quantity))
+            self.Log(f"MarketOrder: {quantity}")
             self.MarketOrder(self.spy, quantity, True)   # async needed for partial fill market orders
 
 # If we want to use methods from other models, you need to inherit from one of them
@@ -90,7 +92,7 @@ class CustomFillModel(ImmediateFillModel):
             absoluteRemaining = absoluteRemaining - absoluteFillQuantity
             self.absoluteRemainingByOrderId[order.Id] = absoluteRemaining
             fill.Status = OrderStatus.PartiallyFilled
-        self.algorithm.Log("CustomFillModel: " + str(fill))
+        self.algorithm.Log(f"CustomFillModel: {fill}")
         return fill
 
 class CustomFeeModel(FeeModel):
@@ -102,7 +104,7 @@ class CustomFeeModel(FeeModel):
         fee = max(1, parameters.Security.Price
                   * parameters.Order.AbsoluteQuantity
                   * 0.00001)
-        self.algorithm.Log("CustomFeeModel: " + str(fee))
+        self.algorithm.Log(f"CustomFeeModel: {fee}")
         return OrderFee(CashAmount(fee, "USD"))
 
 class CustomSlippageModel:
@@ -112,5 +114,15 @@ class CustomSlippageModel:
     def GetSlippageApproximation(self, asset, order):
         # custom slippage math
         slippage = asset.Price * 0.0001 * np.log10(2*float(order.AbsoluteQuantity))
-        self.algorithm.Log("CustomSlippageModel: " + str(slippage))
-        return slippage
+        self.algorithm.Log(f"CustomSlippageModel: {slippage}")
+        return slippage
+
+class CustomBuyingPowerModel(BuyingPowerModel):
+    def __init__(self, algorithm):
+        self.algorithm = algorithm
+
+    def HasSufficientBuyingPowerForOrder(self, parameters):
+        # custom behavior: this model will assume that there is always enough buying power
+        hasSufficientBuyingPowerForOrderResult = HasSufficientBuyingPowerForOrderResult(True)
+        self.algorithm.Log(f"CustomBuyingPowerModel: {hasSufficientBuyingPowerForOrderResult.IsSufficient}")
+        return hasSufficientBuyingPowerForOrderResult

Algorithm.Python/QuantConnect.Algorithm.Python.csproj

@@ -65,6 +65,7 @@
     <Content Include="Alphas\VIXDualThrustAlpha.py" />
     <Content Include="AltData\CachedAlternativeDataAlgorithm.py" />
     <Content Include="AltData\BenzingaNewsAlgorithm.py" />
+    <Content Include="AltData\QuiverWallStreetBetsDataAlgorithm.py" />
     <Content Include="AltData\SECReport8KAlgorithm.py" />
     <Content Include="AltData\SmartInsiderTransactionAlgorithm.py" />
     <Content Include="AltData\USTreasuryYieldCurveRateAlgorithm.py" />
@@ -91,6 +92,7 @@
     <Content Include="CustomConsolidatorRegressionAlgorithm.py" />
     <Content Include="CustomDataAddDataOnSecuritiesChangedRegressionAlgorithm.py" />
     <Content Include="CustomDataAddDataCoarseSelectionRegressionAlgorithm.py" />
+    <None Include="CustomBuyingPowerModelAlgorithm.py" />
     <Content Include="DynamicSecurityDataAlgorithm.py" />
     <Content Include="ConfidenceWeightedFrameworkAlgorithm.py" />
     <Content Include="ExtendedMarketTradingRegressionAlgorithm.py" />

Algorithm.Python/QuantConnect.Algorithm.PythonTools.pyproj

@@ -45,6 +45,7 @@
     <Compile Include="ConstituentsQC500GeneratorAlgorithm.py" />
     <Compile Include="ConstituentsUniverseRegressionAlgorithm.py" />
     <Compile Include="ConvertToFrameworkAlgorithm.py" />
+    <Compile Include="CustomBuyingPowerModelAlgorithm.py" />
     <Compile Include="CustomDataAddDataCoarseSelectionRegressionAlgorithm.py" />
     <Compile Include="CustomDataAddDataOnSecuritiesChangedRegressionAlgorithm.py" />
     <Compile Include="CustomDataAddDataRegressionAlgorithm.py" />

Algorithm.Python/readme.md

@@ -24,16 +24,14 @@ Before we enable python support, follow the [installation instructions](https://
 3. Install [wrapt=1.11.2](https://pypi.org/project/wrapt/) module.
 
 *Note:* If you encounter the "System.DllNotFoundException: python3.6m" runtime error when running Python algorithms on macOS:
-1. Find `libpython3.6m.dylib` in your Python installation folder. If you installed Python with Anaconda, it may be find at
-```
-/Users/{your_user_name}/anaconda3/lib/libpython3.6m.dylib
-```
-2. Open `Lean/Launcher/bin/Debug/Python.Runtime.dll.config`, add the following text and save:
-```
- <configuration>
-    <dllmap dll="python3.6m" target="{the path in step 1 including libpython3.6m.dylib}" os="!windows"/>
-</configuration>
-```
+1. Find `libpython3.6m.dylib` in your Python installation folder. If you installed Python with Anaconda, it may be found at
+    ```
+    /Users/{your_user_name}/anaconda3/lib/libpython3.6m.dylib
+    ```
+2. Open `Lean/Launcher/bin/Debug/Python.Runtime.dll.config`, add the following text under `<configuration> ... </configuration>` and save:
+    ```
+        <dllmap dll="python3.6m" target="{the path in step 1 including libpython3.6m.dylib}" os="osx"/>
+    ```
 Note: Specify the install of v3.6.8 _exactly_, i.e. if with conda `conda install python=3.6.8` as this is a known compatible version and other versions may have issues as of this writing. 
 
 #### [Linux](https://github.com/QuantConnect/Lean#linux-debian-ubuntu)
@@ -50,18 +48,32 @@ conda install -y pandas=0.25.3
 conda install -y wrapt=1.11.2
 ```
 
-*Note:* There is a [known issue](https://github.com/pythonnet/pythonnet/issues/609) with python 3.6.5 that prevents pythonnet installation, please upgrade python to version 3.6.8:
-```
-conda install -y python=3.6.8
-```
+*Note 1:* There is a [known issue](https://github.com/pythonnet/pythonnet/issues/609) with python 3.6.5 that prevents pythonnet installation, please upgrade python to version 3.6.8:
+    ```
+    conda install -y python=3.6.8
+    ```
+    
+*Note 2:* If you encounter the "System.DllNotFoundException: python3.6m" runtime error when running Python algorithms on Linux:
+1. Find `libpython3.6m.so` in your Python installation folder. If you installed Python with Miniconda, it may be found at
+    ```
+    /home/{your_user_name}/miniconda3/envs/{qc_environment}/lib/libpython3.6m.so
+    ```
+   Note that you can create a new virtual environment with all required dependencies by executing:
+   ```
+   conda create -n qc_environment python=3.6.8 cython=0.29.11 pandas=0.25.3 wrapt=1.11.2
 
+   ```
+2. Open `Lean/Launcher/bin/Debug/Python.Runtime.dll.config`, add the following text under `<configuration> ... </configuration>` and save:
+    ```
+        <dllmap dll="python3.6m" target="{the path in step 1 including libpython3.6m.so}" os="linux"/>
+    ```
 ### Run python algorithm
 1. Update the [config](https://github.com/QuantConnect/Lean/blob/master/Launcher/config.json) to run the python algorithm:
-```json
-"algorithm-type-name": "BasicTemplateAlgorithm",
-"algorithm-language": "Python",
-"algorithm-location": "../../../Algorithm.Python/BasicTemplateAlgorithm.py",
-```
+    ```json
+    "algorithm-type-name": "BasicTemplateAlgorithm",
+    "algorithm-language": "Python",
+    "algorithm-location": "../../../Algorithm.Python/BasicTemplateAlgorithm.py",
+    ```
  2. Rebuild LEAN.
  3. Run LEAN. You should see the same result of the C# algorithm you tested earlier.
 

Algorithm/QCAlgorithm.cs

@@ -959,8 +959,19 @@ namespace QuantConnect.Algorithm
             // the time rules need to know the default time zone as well
             TimeRules.SetDefaultTimeZone(timeZone);
 
-            // reset the current time according to the time zone
-            SetDateTime(_startDate.ConvertToUtc(TimeZone));
+            // In BackTest mode we reset the Algorithm time to reflect the new timezone
+            // startDate is set by the user so we expect it to be for their timezone already
+            // so there is no need to update it.
+            if (!LiveMode)
+            {
+                SetDateTime(_startDate.ConvertToUtc(TimeZone));
+            }
+            // In live mode we need to adjust startDate to reflect the new timezone
+            // startDate is set by Lean to the default timezone (New York), so we must update it here
+            else
+            {
+                _startDate = DateTime.UtcNow.ConvertFromUtc(TimeZone).Date;
+            }
         }
 
         /// <summary>
@@ -1362,7 +1373,8 @@ namespace QuantConnect.Algorithm
                 Securities.SetLiveMode(live);
                 if (live)
                 {
-                    _startDate = DateTime.Today;
+                    // startDate is set relative to the algorithm's timezone.
+                    _startDate = DateTime.UtcNow.ConvertFromUtc(TimeZone).Date;
                     _endDate = QuantConnect.Time.EndOfTime;
                 }
             }

Api/Api.cs

@@ -19,6 +19,7 @@ using System.IO;
 using System.Linq;
 using System.Net;
 using Newtonsoft.Json;
+using Newtonsoft.Json.Linq;
 using QuantConnect.Interfaces;
 using QuantConnect.Orders;
 using RestSharp;
@@ -74,13 +75,15 @@ namespace QuantConnect.Api
 
         public ProjectResponse CreateProject(string name, Language language)
         {
-            var request = new RestRequest("projects/create", Method.POST);
+            var request = new RestRequest("projects/create", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.RequestFormat = DataFormat.Json;
             request.AddParameter("application/json", JsonConvert.SerializeObject(new
             {
-                name = name,
-                language = language
+                name, 
+                language
             }), ParameterType.RequestBody);
 
             ProjectResponse result;
@@ -96,10 +99,15 @@ namespace QuantConnect.Api
 
         public ProjectResponse ReadProject(int projectId)
         {
-            var request = new RestRequest("projects/read", Method.GET);
+            var request = new RestRequest("projects/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("projectId", projectId);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId
+                }), ParameterType.RequestBody);
 
             ProjectResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -113,8 +121,11 @@ namespace QuantConnect.Api
 
         public ProjectResponse ListProjects()
         {
-            var request = new RestRequest("projects/read", Method.GET);
-            request.RequestFormat = DataFormat.Json;
+            var request = new RestRequest("projects/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
             ProjectResponse result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -131,11 +142,17 @@ namespace QuantConnect.Api
 
         public ProjectFilesResponse AddProjectFile(int projectId, string name, string content)
         {
-            var request = new RestRequest("files/create", Method.POST);
+            var request = new RestRequest("files/create", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("name", name);
-            request.AddParameter("content", content);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId,
+                    name,
+                    content
+                }), ParameterType.RequestBody);
 
             ProjectFilesResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -153,11 +170,17 @@ namespace QuantConnect.Api
 
         public RestResponse UpdateProjectFileName(int projectId, string oldFileName, string newFileName)
         {
-            var request = new RestRequest("files/update", Method.POST);
+            var request = new RestRequest("files/update", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("name", oldFileName);
-            request.AddParameter("newName", newFileName);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId,
+                    name = oldFileName,
+                    newName = newFileName
+                }), ParameterType.RequestBody);
 
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -175,11 +198,17 @@ namespace QuantConnect.Api
 
         public RestResponse UpdateProjectFileContent(int projectId, string fileName, string newFileContents)
         {
-            var request = new RestRequest("files/update", Method.POST);
+            var request = new RestRequest("files/update", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("name", fileName);
-            request.AddParameter("content", newFileContents);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId,
+                    name = fileName,
+                    content = newFileContents
+                }), ParameterType.RequestBody);
 
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -195,9 +224,15 @@ namespace QuantConnect.Api
 
         public ProjectFilesResponse ReadProjectFiles(int projectId)
         {
-            var request = new RestRequest("files/read", Method.GET);
+            var request = new RestRequest("files/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("projectId", projectId);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId
+                }), ParameterType.RequestBody);
 
             ProjectFilesResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -214,10 +249,16 @@ namespace QuantConnect.Api
 
         public ProjectFilesResponse ReadProjectFile(int projectId, string fileName)
         {
-            var request = new RestRequest("files/read", Method.GET);
+            var request = new RestRequest("files/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("name", fileName);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId,
+                    name = fileName
+                }), ParameterType.RequestBody);
 
             ProjectFilesResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -233,10 +274,16 @@ namespace QuantConnect.Api
 
         public RestResponse DeleteProjectFile(int projectId, string name)
         {
-            var request = new RestRequest("files/delete", Method.POST);
+            var request = new RestRequest("files/delete", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("name", name);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId,
+                    name,
+                }), ParameterType.RequestBody);
 
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -251,12 +298,16 @@ namespace QuantConnect.Api
 
         public RestResponse DeleteProject(int projectId)
         {
-            var request = new RestRequest("projects/delete", Method.POST);
-            request.RequestFormat = DataFormat.Json;
+            var request = new RestRequest("projects/delete", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
             request.AddParameter("application/json", JsonConvert.SerializeObject(new
             {
-                projectId = projectId
+                projectId
             }), ParameterType.RequestBody);
+
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -270,11 +321,16 @@ namespace QuantConnect.Api
 
         public Compile CreateCompile(int projectId)
         {
-            var request = new RestRequest("compile/create", Method.POST);
+            var request = new RestRequest("compile/create", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
             request.AddParameter("application/json", JsonConvert.SerializeObject(new
             {
-                projectId = projectId
+                projectId
             }), ParameterType.RequestBody);
+
             Compile result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -289,10 +345,17 @@ namespace QuantConnect.Api
 
         public Compile ReadCompile(int projectId, string compileId)
         {
-            var request = new RestRequest("compile/read", Method.GET);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("compileId", compileId);
+            var request = new RestRequest("compile/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                projectId,
+                compileId
+            }), ParameterType.RequestBody);
+
             Compile result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -309,10 +372,18 @@ namespace QuantConnect.Api
 
         public Backtest CreateBacktest(int projectId, string compileId, string backtestName)
         {
-            var request = new RestRequest("backtests/create", Method.POST);
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("compileId", compileId);
-            request.AddParameter("backtestName", backtestName);
+            var request = new RestRequest("backtests/create", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                projectId,
+                compileId,
+                backtestName
+            }), ParameterType.RequestBody);
+
             Backtest result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -327,9 +398,17 @@ namespace QuantConnect.Api
 
         public Backtest ReadBacktest(int projectId, string backtestId)
         {
-            var request = new RestRequest("backtests/read", Method.GET);
-            request.AddParameter("backtestId", backtestId);
-            request.AddParameter("projectId", projectId);
+            var request = new RestRequest("backtests/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                projectId,
+                backtestId
+            }), ParameterType.RequestBody);
+
             Backtest result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -346,15 +425,19 @@ namespace QuantConnect.Api
 
         public RestResponse UpdateBacktest(int projectId, string backtestId, string name = "", string note = "")
         {
-            var request = new RestRequest("backtests/update", Method.POST);
-            request.RequestFormat = DataFormat.Json;
+            var request = new RestRequest("backtests/update", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
             request.AddParameter("application/json", JsonConvert.SerializeObject(new
             {
-                projectId = projectId,
-                backtestId = backtestId,
-                name = name,
-                note = note
+                projectId,
+                backtestId,
+                name,
+                note
             }), ParameterType.RequestBody);
+
             Backtest result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -368,8 +451,16 @@ namespace QuantConnect.Api
 
         public BacktestList ListBacktests(int projectId)
         {
-            var request = new RestRequest("backtests/read", Method.GET);
-            request.AddParameter("projectId", projectId);
+            var request = new RestRequest("backtests/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                projectId,
+            }), ParameterType.RequestBody);
+
             BacktestList result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -384,10 +475,17 @@ namespace QuantConnect.Api
 
         public RestResponse DeleteBacktest(int projectId, string backtestId)
         {
-            var request = new RestRequest("backtests/delete", Method.POST);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("backtestId", backtestId);
-            request.AddParameter("projectId", projectId);
+            var request = new RestRequest("backtests/delete", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                projectId,
+                backtestId
+            }), ParameterType.RequestBody);
+
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -398,7 +496,7 @@ namespace QuantConnect.Api
         /// </summary>
         /// <param name="projectId">Id of the project on QuantConnect</param>
         /// <param name="compileId">Id of the compilation on QuantConnect</param>
-        /// <param name="serverType">Type of server instance that will run the algorithm</param>
+        /// <param name="nodeId">Id of the node that will run the algorithm</param>
         /// <param name="baseLiveAlgorithmSettings">Brokerage specific <see cref="BaseLiveAlgorithmSettings">BaseLiveAlgorithmSettings</see>.</param>
         /// <param name="versionId">The version of the Lean used to run the algorithm.
         ///                         -1 is master, however, sometimes this can create problems with live deployments.
@@ -407,19 +505,23 @@ namespace QuantConnect.Api
 
         public LiveAlgorithm CreateLiveAlgorithm(int projectId,
                                                  string compileId,
-                                                 string serverType,
+                                                 string nodeId,
                                                  BaseLiveAlgorithmSettings baseLiveAlgorithmSettings,
                                                  string versionId = "-1")
         {
-            var request = new RestRequest("live/create", Method.POST);
-            request.AddHeader("Accept", "application/json");
-            request.Parameters.Clear();
-            var body = JsonConvert.SerializeObject(new LiveAlgorithmApiSettingsWrapper(projectId,
-                                                                                       compileId,
-                                                                                       serverType,
-                                                                                       baseLiveAlgorithmSettings,
-                                                                                       versionId));
-            request.AddParameter("application/json", body, ParameterType.RequestBody);
+            var request = new RestRequest("live/create", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(
+                new LiveAlgorithmApiSettingsWrapper
+                (projectId,
+                compileId,
+                nodeId,
+                baseLiveAlgorithmSettings,
+                versionId)
+                ), ParameterType.RequestBody);
 
             LiveAlgorithm result;
             ApiConnection.TryRequest(request, out result);
@@ -449,18 +551,26 @@ namespace QuantConnect.Api
                     "The Api only supports Algorithm Statuses of Running, Stopped, RuntimeError and Liquidated");
             }
 
-            var request = new RestRequest("live/read", Method.GET);
-
-            if (status.HasValue)
+            var request = new RestRequest("live/read", Method.POST)
             {
-                request.AddParameter("status", status.ToString());
-            }
+                RequestFormat = DataFormat.Json
+            };
 
             var epochStartTime = startTime == null ? 0 : Time.DateTimeToUnixTimeStamp(startTime.Value);
             var epochEndTime   = endTime   == null ? Time.DateTimeToUnixTimeStamp(DateTime.UtcNow) : Time.DateTimeToUnixTimeStamp(endTime.Value);
 
-            request.AddParameter("start", epochStartTime);
-            request.AddParameter("end", epochEndTime);
+            JObject obj = new JObject
+            {
+                { "start", epochStartTime },
+                { "end", epochEndTime }
+            };
+
+            if (status.HasValue)
+            {
+                obj.Add("status", status.ToString());
+            }
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(obj), ParameterType.RequestBody);
 
             LiveList result;
             ApiConnection.TryRequest(request, out result);
@@ -476,9 +586,17 @@ namespace QuantConnect.Api
 
         public LiveAlgorithmResults ReadLiveAlgorithm(int projectId, string deployId)
         {
-            var request = new RestRequest("live/read", Method.GET);
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("deployId", deployId);
+            var request = new RestRequest("live/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId,
+                    deployId
+                }), ParameterType.RequestBody);
+
             LiveAlgorithmResults result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -492,9 +610,16 @@ namespace QuantConnect.Api
 
         public RestResponse LiquidateLiveAlgorithm(int projectId)
         {
-            var request = new RestRequest("live/update/liquidate", Method.POST);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("projectId", projectId);
+            var request = new RestRequest("live/update/liquidate", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+                {
+                    projectId
+                }), ParameterType.RequestBody);
+
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -508,9 +633,16 @@ namespace QuantConnect.Api
 
         public RestResponse StopLiveAlgorithm(int projectId)
         {
-            var request = new RestRequest("live/update/stop", Method.POST);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("projectId", projectId);
+            var request = new RestRequest("live/update/stop", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                projectId
+            }), ParameterType.RequestBody);
+
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
             return result;
@@ -530,13 +662,19 @@ namespace QuantConnect.Api
             var epochStartTime = startTime == null ? 0 : Time.DateTimeToUnixTimeStamp(startTime.Value);
             var epochEndTime   = endTime   == null ? Time.DateTimeToUnixTimeStamp(DateTime.UtcNow) : Time.DateTimeToUnixTimeStamp(endTime.Value);
 
-            var request = new RestRequest("live/read/log", Method.GET);
+            var request = new RestRequest("live/read/log", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("format", "json");
-            request.AddParameter("projectId", projectId);
-            request.AddParameter("algorithmId", algorithmId);
-            request.AddParameter("start", epochStartTime);
-            request.AddParameter("end", epochEndTime);
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                format = "json",
+                projectId,
+                algorithmId,
+                start = epochStartTime,
+                end = epochEndTime
+            }), ParameterType.RequestBody);
 
             LiveLog result;
             ApiConnection.TryRequest(request, out result);
@@ -553,14 +691,20 @@ namespace QuantConnect.Api
 
         public Link ReadDataLink(Symbol symbol, Resolution resolution, DateTime date)
         {
-            var request = new RestRequest("data/read", Method.GET);
+            var request = new RestRequest("data/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
 
-            request.AddParameter("format", "link");
-            request.AddParameter("ticker", symbol.Value.ToLowerInvariant());
-            request.AddParameter("type", symbol.ID.SecurityType.ToLower());
-            request.AddParameter("market", symbol.ID.Market);
-            request.AddParameter("resolution", resolution);
-            request.AddParameter("date", date.ToStringInvariant("yyyyMMdd"));
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                format = "link",
+                ticker = symbol.Value.ToLowerInvariant(),
+                type = symbol.ID.SecurityType.ToLower(),
+                market = symbol.ID.Market,
+                resolution = resolution.ToString(),
+                date = date.ToStringInvariant("yyyyMMdd")
+            }), ParameterType.RequestBody);
 
             Link result;
             ApiConnection.TryRequest(request, out result);
@@ -575,44 +719,22 @@ namespace QuantConnect.Api
         /// <returns><see cref="BacktestReport"/></returns>
         public BacktestReport ReadBacktestReport(int projectId, string backtestId)
         {
-            var request = new RestRequest("backtests/read/report", Method.POST);
-            request.AddParameter("backtestId", backtestId);
-            request.AddParameter("projectId", projectId);
+            var request = new RestRequest("backtests/read/report", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                backtestId,
+                projectId
+            }), ParameterType.RequestBody);
 
             BacktestReport report;
             ApiConnection.TryRequest(request, out report);
             return report;
         }
 
-        /// <summary>
-        /// Will get the prices for requested symbols
-        /// </summary>
-        /// <param name="symbols">Symbols for which the price is requested</param>
-        /// <returns><see cref="Prices"/></returns>
-        public PricesList ReadPrices(IEnumerable<Symbol> symbols)
-        {
-            var symbolByID = new Dictionary<string, Symbol>();
-            foreach (var symbol in symbols)
-            {
-                symbolByID[symbol.ID.ToString()] = symbol;
-            }
-
-            var request = new RestRequest("prices", Method.POST);
-            var symbolsToRequest = string.Join(",", symbolByID.Keys);
-            request.AddParameter("symbols", symbolsToRequest);
-
-            PricesList pricesList;
-            if (ApiConnection.TryRequest(request, out pricesList))
-            {
-                foreach (var price in pricesList.Prices)
-                {
-                    price.Symbol = symbolByID[price.SymbolID];
-                }
-            }
-
-            return pricesList;
-        }
-
         /// <summary>
         /// Method to download and save the data purchased through QuantConnect
         /// </summary>
@@ -703,54 +825,6 @@ namespace QuantConnect.Api
             //
         }
 
-        /// <summary>
-        /// Gets all split events between the specified times. From and to are inclusive.
-        /// </summary>
-        /// <param name="from">The first date to get splits for</param>
-        /// <param name="to">The last date to get splits for</param>
-        /// <returns>A list of all splits in the specified range</returns>
-        public List<Data.Market.Split> GetSplits(DateTime from, DateTime to)
-        {
-            var request = new RestRequest("splits", Method.POST);
-            request.AddParameter("from", from.ToStringInvariant("yyyyMMdd"));
-            request.AddParameter("to", from.ToStringInvariant("yyyyMMdd"));
-
-            SplitList splits;
-            ApiConnection.TryRequest(request, out splits);
-
-            return splits.Splits.Select(s => new Data.Market.Split(
-                s.Symbol,
-                s.Date,
-                s.ReferencePrice,
-                s.SplitFactor,
-                SplitType.SplitOccurred)
-            ).ToList();
-        }
-
-        /// <summary>
-        /// Gets all dividend events between the specified times. From and to are inclusive.
-        /// </summary>
-        /// <param name="from">The first date to get dividend for</param>
-        /// <param name="to">The last date to get dividend for</param>
-        /// <returns>A list of all dividend in the specified range</returns>
-        public List<Data.Market.Dividend> GetDividends(DateTime from, DateTime to)
-        {
-            var request = new RestRequest("dividends", Method.POST);
-            request.AddParameter("from", from.ToStringInvariant("yyyyMMdd"));
-            request.AddParameter("to", from.ToStringInvariant("yyyyMMdd"));
-
-            DividendList dividends;
-            ApiConnection.TryRequest(request, out dividends);
-
-            return dividends.Dividends.Select(s => new Data.Market.Dividend(
-                s.Symbol,
-                s.Date,
-                s.DividendPerShare,
-                s.ReferencePrice)
-            ).ToList();
-        }
-
-
         /// <summary>
         /// Local implementation for downloading data to algorithms
         /// </summary>
@@ -810,14 +884,20 @@ namespace QuantConnect.Api
         /// <see cref="Node"/></returns>
         public CreatedNode CreateNode(string name, string organizationId, SKU sku)
         {
-            var request = new RestRequest("nodes/create", Method.POST);
-            request.AddParameter("name", name);
-            request.AddParameter("organizationId", organizationId);
-            request.AddParameter("sku", sku.ToString());
+            var request = new RestRequest("nodes/create", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                name,
+                organizationId,
+                sku = sku.ToString()
+            }), ParameterType.RequestBody);
 
             CreatedNode result;
             ApiConnection.TryRequest(request, out result);
-
             return result;
         }
 
@@ -829,9 +909,15 @@ namespace QuantConnect.Api
         /// <returns><see cref="NodeList"/> containing Backtest, Research, and Live Nodes</returns>
         public NodeList ReadNodes(string organizationId)
         {
-            var request = new RestRequest("nodes/read", Method.POST);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("organizationId", organizationId);
+            var request = new RestRequest("nodes/read", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                organizationId,
+            }), ParameterType.RequestBody);
 
             NodeList result;
             ApiConnection.TryRequest(request, out result);
@@ -847,11 +933,17 @@ namespace QuantConnect.Api
         /// <returns><see cref="RestResponse"/> containing success response and errors</returns>
         public RestResponse UpdateNode(string nodeId, string newName, string organizationId)
         {
-            var request = new RestRequest("nodes/update", Method.POST);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("nodeId", nodeId);
-            request.AddParameter("name", newName);
-            request.AddParameter("organizationId", organizationId);
+            var request = new RestRequest("nodes/update", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                nodeId,
+                name = newName,
+                organizationId
+            }), ParameterType.RequestBody);
 
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -866,10 +958,16 @@ namespace QuantConnect.Api
         /// <returns><see cref="RestResponse"/> containing success response and errors</returns>
         public RestResponse DeleteNode(string nodeId, string organizationId)
         {
-            var request = new RestRequest("nodes/delete", Method.POST);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("nodeId", nodeId);
-            request.AddParameter("organizationId", organizationId);
+            var request = new RestRequest("nodes/delete", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                nodeId,
+                organizationId
+            }), ParameterType.RequestBody);
 
             RestResponse result;
             ApiConnection.TryRequest(request, out result);
@@ -884,10 +982,16 @@ namespace QuantConnect.Api
         /// <returns><see cref="RestResponse"/> containing success response and errors</returns>
         public RestResponse StopNode(string nodeId, string organizationId)
         {
-            var request = new RestRequest("nodes/stop", Method.POST);
-            request.RequestFormat = DataFormat.Json;
-            request.AddParameter("nodeId", nodeId);
-            request.AddParameter("organizationId", organizationId);
+            var request = new RestRequest("nodes/stop", Method.POST)
+            {
+                RequestFormat = DataFormat.Json
+            };
+
+            request.AddParameter("application/json", JsonConvert.SerializeObject(new
+            {
+                nodeId,
+                organizationId
+            }), ParameterType.RequestBody);
 
             RestResponse result;
             ApiConnection.TryRequest(request, out result);

Brokerages/Alpaca/AlpacaBrokerage.cs

@@ -110,6 +110,11 @@ namespace QuantConnect.Brokerages.Alpaca
         /// </summary>
         public override bool IsConnected => _sockClient.IsConnected;
 
+        /// <summary>
+        /// Returns the brokerage account's base currency
+        /// </summary>
+        public override string AccountBaseCurrency => Currencies.USD;
+
         /// <summary>
         /// Connects the client to the broker's remote servers
         /// </summary>

Brokerages/ApiPriceProvider.cs

@@ -54,19 +54,8 @@ namespace QuantConnect.Brokerages
         /// <returns>The latest price</returns>
         public decimal GetLastPrice(Symbol symbol)
         {
-            var result = _api.ReadPrices(new[] { symbol });
-            if (!result.Success)
-            {
-                throw new Exception($"ReadPrices error: {string.Join(" - ", result.Errors)}");
-            }
-
-            var priceData = result.Prices.FirstOrDefault(x => x.Symbol == symbol);
-            if (priceData == null)
-            {
-                throw new Exception($"No price data available for symbol: {symbol.Value}");
-            }
-
-            return priceData.Price;
+            //NOP ReadPrices endpoint has been removed
+            throw new InvalidOperationException("Prices endpoint is no longer supported");
         }
     }
 }

Brokerages/Binance/BinanceBrokerage.cs

@@ -117,11 +117,6 @@ namespace QuantConnect.Brokerages.Binance
         /// </summary>
         public override bool IsConnected => WebSocket.IsOpen;
 
-        /// <summary>
-        /// Returns the brokerage account's base currency
-        /// </summary>
-        public override string AccountBaseCurrency => "USDT";
-
         /// <summary>
         /// Creates wss connection
         /// </summary>

Brokerages/Bitfinex/BitfinexSubscriptionManager.cs

@@ -51,10 +51,10 @@ namespace QuantConnect.Brokerages.Bitfinex
         private readonly BitfinexSymbolMapper _symbolMapper;
         private readonly RateGate _connectionRateLimiter = new RateGate(5, TimeSpan.FromMinutes(1));
         private readonly ConcurrentDictionary<Symbol, List<BitfinexWebSocketWrapper>> _subscriptionsBySymbol = new ConcurrentDictionary<Symbol, List<BitfinexWebSocketWrapper>>();
-        private readonly ConcurrentDictionary<BitfinexWebSocketWrapper, List<Channel>> _channelsByWebSocket = new ConcurrentDictionary<BitfinexWebSocketWrapper, List<Channel>>();
-        private readonly ConcurrentDictionary<int, Channel> _channels = new ConcurrentDictionary<int, Channel>();
+        private readonly ConcurrentDictionary<BitfinexWebSocketWrapper, BitfinexWebSocketChannels> _channelsByWebSocket = new ConcurrentDictionary<BitfinexWebSocketWrapper, BitfinexWebSocketChannels>();
         private readonly ConcurrentDictionary<Symbol, DefaultOrderBook> _orderBooks = new ConcurrentDictionary<Symbol, DefaultOrderBook>();
-        private readonly IReadOnlyDictionary<TickType, string> _tickType2ChannelName = new Dictionary<TickType, string>() {
+        private readonly IReadOnlyDictionary<TickType, string> _tickType2ChannelName = new Dictionary<TickType, string>
+        {
             { TickType.Trade, "trades"},
             { TickType.Quote, "book"}
         };
@@ -101,7 +101,7 @@ namespace QuantConnect.Brokerages.Bitfinex
                             return v;
                         });
 
-                    Log.Trace($"BitfinexBrokerage.Subscribe(): Sent subscribe for {symbol.Value}.");
+                    Log.Trace($"BitfinexBrokerage.Subscribe(): Sent subscribe for {symbol.Value}/{tickType}.");
 
                     if (_onSubscribeEvent.WaitOne(TimeSpan.FromSeconds(10)) && _subscribeErrorCode == 0)
                     {
@@ -109,7 +109,7 @@ namespace QuantConnect.Brokerages.Bitfinex
                     }
                     else
                     {
-                        Log.Trace($"BitfinexBrokerage.Subscribe(): Could not subscribe to {symbol.Value}.");
+                        Log.Trace($"BitfinexBrokerage.Subscribe(): Could not subscribe to {symbol.Value}/{tickType}.");
                         states.Add(false);
                     }
                 }
@@ -130,24 +130,24 @@ namespace QuantConnect.Brokerages.Bitfinex
         /// <param name="tickType">Type of tick data</param>
         protected override bool Unsubscribe(IEnumerable<Symbol> symbols, TickType tickType)
         {
-            string channelName = ChannelNameFromTickType(tickType);
+            var channelName = ChannelNameFromTickType(tickType);
             var states = new List<bool>(symbols.Count());
             foreach (var symbol in symbols)
             {
                 List<BitfinexWebSocketWrapper> subscriptions;
                 if (_subscriptionsBySymbol.TryGetValue(symbol, out subscriptions))
                 {
-                    for (int i = subscriptions.Count - 1; i >= 0; i--)
+                    for (var i = subscriptions.Count - 1; i >= 0; i--)
                     {
                         var webSocket = subscriptions[i];
                         _onUnsubscribeEvent.Reset();
                         try
                         {
-                            Channel channel = new Channel(channelName, symbol);
-                            List<Channel> channels;
+                            var channel = new Channel(channelName, symbol);
+                            BitfinexWebSocketChannels channels;
                             if (_channelsByWebSocket.TryGetValue(webSocket, out channels) && channels.Contains(channel))
                             {
-                                UnsubscribeChannel(webSocket, channel);
+                                UnsubscribeChannel(webSocket, channels, channel);
 
                                 if (_onUnsubscribeEvent.WaitOne(TimeSpan.FromSeconds(30)))
                                 {
@@ -199,9 +199,10 @@ namespace QuantConnect.Brokerages.Bitfinex
             return webSocket;
         }
 
-        private void UnsubscribeChannel(IWebSocket webSocket, Channel channel)
+        private void UnsubscribeChannel(IWebSocket webSocket, BitfinexWebSocketChannels channels, Channel channel)
         {
-            int channelId = _channels.First(c => c.Value.Equals(channel)).Key;
+            var channelId = channels.GetChannelId(channel);
+
             webSocket.Send(JsonConvert.SerializeObject(new
             {
                 @event = "unsubscribe",
@@ -211,19 +212,12 @@ namespace QuantConnect.Brokerages.Bitfinex
 
         private BitfinexWebSocketWrapper GetFreeWebSocket(Channel channel)
         {
-            int count;
-
             lock (_locker)
             {
                 foreach (var kvp in _channelsByWebSocket)
                 {
                     if (kvp.Value.Count < MaximumSubscriptionsPerSocket)
                     {
-                        kvp.Value.Add(channel);
-
-                        count = _channelsByWebSocket.Sum(x => x.Value.Count);
-                        Log.Trace($"BitfinexSubscriptionManager.GetFreeWebSocket(): Channel added: Total channels:{count}");
-
                         return kvp.Key;
                     }
                 }
@@ -242,10 +236,7 @@ namespace QuantConnect.Brokerages.Bitfinex
 
             lock (_locker)
             {
-                _channelsByWebSocket.TryAdd(webSocket, new List<Channel> { channel });
-
-                count = _channelsByWebSocket.Sum(x => x.Value.Count);
-                Log.Trace($"BitfinexSubscriptionManager.GetFreeWebSocket(): Channel added: Total channels:{count}");
+                _channelsByWebSocket.TryAdd(webSocket, new BitfinexWebSocketChannels());
             }
 
             webSocket.Initialize(_wssUrl);
@@ -332,7 +323,7 @@ namespace QuantConnect.Brokerages.Bitfinex
 
             Log.Trace($"BitfinexSubscriptionManager.OnReconnectRequested(): Reconnected: IsOpen:{webSocket.IsOpen} [Id: {connectionHandler.ConnectionId}]");
 
-            List<Channel> channels;
+            BitfinexWebSocketChannels channels;
             lock (_locker)
             {
                 if (!_channelsByWebSocket.TryGetValue(webSocket, out channels))
@@ -343,7 +334,7 @@ namespace QuantConnect.Brokerages.Bitfinex
 
             Log.Trace($"BitfinexSubscriptionManager.OnReconnectRequested(): Resubscribing channels. [Id: {connectionHandler.ConnectionId}]");
 
-            foreach (var channel in channels)
+            foreach (var channel in channels.Values)
             {
                 webSocket.Send(JsonConvert.SerializeObject(new
                 {
@@ -380,7 +371,7 @@ namespace QuantConnect.Brokerages.Bitfinex
 
                             // trade execution
                             case "te":
-                                OnUpdate(channel, token[2].ToObject<string[]>());
+                                OnUpdate(webSocket, channel, token[2].ToObject<string[]>());
                                 break;
 
                             // ignored -- trades already handled in "te" message
@@ -400,6 +391,7 @@ namespace QuantConnect.Brokerages.Bitfinex
                         if (token[1][0].Type == JTokenType.Array)
                         {
                             OnSnapshot(
+                                webSocket,
                                 channel,
                                 token[1].ToObject<string[][]>()
                             );
@@ -408,6 +400,7 @@ namespace QuantConnect.Brokerages.Bitfinex
                         {
                             // pass channel id as separate arg
                             OnUpdate(
+                                webSocket,
                                 channel,
                                 token[1].ToObject<string[]>()
                             );
@@ -416,7 +409,7 @@ namespace QuantConnect.Brokerages.Bitfinex
                 }
                 else if (token is JObject)
                 {
-                    var raw = token.ToObject<Messages.BaseMessage>();
+                    var raw = token.ToObject<BaseMessage>();
                     switch (raw.Event.ToLowerInvariant())
                     {
                         case "subscribed":
@@ -457,7 +450,7 @@ namespace QuantConnect.Brokerages.Bitfinex
             }
         }
 
-        private void OnSubscribe(BitfinexWebSocketWrapper webSocket, Messages.ChannelSubscription data)
+        private void OnSubscribe(BitfinexWebSocketWrapper webSocket, ChannelSubscription data)
         {
             try
             {
@@ -465,7 +458,17 @@ namespace QuantConnect.Brokerages.Bitfinex
                 {
                     var channel = new Channel(data.Channel, _symbolMapper.GetLeanSymbol(data.Symbol));
 
-                    _channels.AddOrUpdate(data.ChannelId, channel);
+                    BitfinexWebSocketChannels channels;
+                    if (!_channelsByWebSocket.TryGetValue(webSocket, out channels))
+                    {
+                        _onSubscribeEvent.Set();
+                        return;
+                    }
+
+                    channels.TryAdd(data.ChannelId, channel);
+
+                    Log.Trace($"BitfinexSubscriptionManager.OnSubscribe(): Channel subscribed: Id:{data.ChannelId} {channel.Symbol}/{channel.Name}");
+
                     _onSubscribeEvent.Set();
 
                     webSocket.ConnectionHandler.EnableMonitoring(true);
@@ -478,23 +481,27 @@ namespace QuantConnect.Brokerages.Bitfinex
             }
         }
 
-        private void OnUnsubscribe(BitfinexWebSocketWrapper webSocket, Messages.ChannelUnsubscribing data)
+        private void OnUnsubscribe(BitfinexWebSocketWrapper webSocket, ChannelUnsubscribing data)
         {
             try
             {
                 lock (_locker)
                 {
+                    BitfinexWebSocketChannels channels;
+                    if (!_channelsByWebSocket.TryGetValue(webSocket, out channels))
+                    {
+                        return;
+                    }
+
                     Channel channel;
-                    if (!_channels.TryRemove(data.ChannelId, out channel)) return;
+                    if (!channels.TryRemove(data.ChannelId, out channel))
+                    {
+                        return;
+                    }
 
                     _onUnsubscribeEvent.Set();
 
-                    List<Channel> channels;
-                    if (!_channelsByWebSocket.TryGetValue(webSocket, out channels)) return;
-
-                    channels.Remove(channel);
-
-                    if (channels.Count(c => c.Symbol.Equals(channel.Symbol)) == 0)
+                    if (channels.Values.Count(c => c.Symbol.Equals(channel.Symbol)) == 0)
                     {
                         List<BitfinexWebSocketWrapper> subscriptions;
                         if (_subscriptionsBySymbol.TryGetValue(channel.Symbol, out subscriptions))
@@ -508,7 +515,10 @@ namespace QuantConnect.Brokerages.Bitfinex
                         }
                     }
 
-                    if (channels.Count != 0) return;
+                    if (channels.Count != 0)
+                    {
+                        return;
+                    }
 
                     _channelsByWebSocket.TryRemove(webSocket, out channels);
                 }
@@ -523,7 +533,7 @@ namespace QuantConnect.Brokerages.Bitfinex
             }
         }
 
-        private void OnSnapshot(int channelId, string[][] entries)
+        private void OnSnapshot(BitfinexWebSocketWrapper webSocket, int channelId, string[][] entries)
         {
             try
             {
@@ -531,7 +541,13 @@ namespace QuantConnect.Brokerages.Bitfinex
 
                 lock (_locker)
                 {
-                    if (!_channels.TryGetValue(channelId, out channel))
+                    BitfinexWebSocketChannels channels;
+                    if (!_channelsByWebSocket.TryGetValue(webSocket, out channels))
+                    {
+                        return;
+                    }
+
+                    if (!channels.TryGetValue(channelId, out channel))
                     {
                         _brokerage.OnMessage(new BrokerageMessageEvent(BrokerageMessageType.Warning, -1, $"Message received from unknown channel Id {channelId}"));
                         return;
@@ -592,7 +608,7 @@ namespace QuantConnect.Brokerages.Bitfinex
             }
         }
 
-        private void OnUpdate(int channelId, string[] entries)
+        private void OnUpdate(BitfinexWebSocketWrapper webSocket, int channelId, string[] entries)
         {
             try
             {
@@ -600,7 +616,13 @@ namespace QuantConnect.Brokerages.Bitfinex
 
                 lock (_locker)
                 {
-                    if (!_channels.TryGetValue(channelId, out channel))
+                    BitfinexWebSocketChannels channels;
+                    if (!_channelsByWebSocket.TryGetValue(webSocket, out channels))
+                    {
+                        return;
+                    }
+
+                    if (!channels.TryGetValue(channelId, out channel))
                     {
                         _brokerage.OnMessage(new BrokerageMessageEvent(BrokerageMessageType.Warning, -1, $"Message received from unknown channel Id {channelId}"));
                         return;

Brokerages/Bitfinex/BitfinexWebSocketChannels.cs

@@ -0,0 +1,47 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using System.Collections.Concurrent;
+using System.Linq;
+using QuantConnect.Data;
+
+namespace QuantConnect.Brokerages.Bitfinex
+{
+    /// <summary>
+    /// Contains the channel mappings for a WebSocket connection
+    /// </summary>
+    public class BitfinexWebSocketChannels : ConcurrentDictionary<int, Channel>
+    {
+        /// <summary>
+        /// Determines whether the dictionary contains a specific channel.
+        /// </summary>
+        /// <param name="channel">The channel</param>
+        /// <returns>true if the channel was found</returns>
+        public bool Contains(Channel channel)
+        {
+            return Values.Contains(channel);
+        }
+
+        /// <summary>
+        /// Returns the channel id for the given channel.
+        /// </summary>
+        /// <param name="channel">The channel</param>
+        /// <returns>The channel id</returns>
+        public int GetChannelId(Channel channel)
+        {
+            return this.First(c => c.Value.Equals(channel)).Key;
+        }
+    }
+}

Brokerages/Brokerage.cs

@@ -226,7 +226,7 @@ namespace QuantConnect.Brokerages
         /// <summary>
         /// Returns the brokerage account's base currency
         /// </summary>
-        public virtual string AccountBaseCurrency { get; protected set; } = Currencies.USD;
+        public virtual string AccountBaseCurrency { get; protected set; }
 
         /// <summary>
         /// Gets the history for the requested security

Brokerages/Fxcm/FxcmBrokerage.Messaging.cs

@@ -107,7 +107,8 @@ namespace QuantConnect.Brokerages.Fxcm
             AutoResetEvent autoResetEvent;
             lock (_locker)
             {
-                _currentRequest = _gateway.requestOpenOrders(_accountId);
+                _currentRequest = _gateway.requestOpenOrders(null);
+
                 autoResetEvent = new AutoResetEvent(false);
                 _mapRequestsToAutoResetEvents[_currentRequest] = autoResetEvent;
             }

Brokerages/GDAX/GDAXBrokerage.Messaging.cs

@@ -539,7 +539,12 @@ namespace QuantConnect.Brokerages.GDAX
 
                         if (response.StatusCode != HttpStatusCode.OK)
                         {
-                            throw new Exception($"GDAXBrokerage.FillMonitorAction(): request failed: [{(int)response.StatusCode}] {response.StatusDescription}, Content: {response.Content}, ErrorMessage: {response.ErrorMessage}");
+                            OnMessage(new BrokerageMessageEvent(
+                                BrokerageMessageType.Warning,
+                                -1,
+                                $"GDAXBrokerage.FillMonitorAction(): request failed: [{(int)response.StatusCode}] {response.StatusDescription}, Content: {response.Content}, ErrorMessage: {response.ErrorMessage}"));
+
+                            continue;
                         }
 
                         var fills = JsonConvert.DeserializeObject<List<Messages.Fill>>(response.Content);

Brokerages/InteractiveBrokers/IB-symbol-map.json

@@ -14,5 +14,6 @@
   "MXP": "6M",
   "RUR": "6R",
   "ZAR": "6Z",
-  "BRR": "BTC"
+  "BRR": "BTC",
+  "DA": "DC"
 }

Brokerages/InteractiveBrokers/InteractiveBrokersBrokerage.cs

@@ -54,14 +54,10 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
         // Existing orders created in TWS can *only* be cancelled/modified when connected with ClientId = 0
         private const int ClientId = 0;
 
-        // next valid order id for this client
+        // next valid order id (or request id, or ticker id) for this client
         private int _nextValidId;
         private readonly object _nextValidIdLocker = new object();
 
-        // next valid request id for queries
-        private int _nextRequestId;
-        private int _nextTickerId;
-
         private readonly int _port;
         private readonly string _account;
         private readonly string _host;
@@ -399,7 +395,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
                 {
                     var orderId = Parse.Int(id);
 
-                    _requestInformation[orderId] = "CancelOrder: " + order;
+                    _requestInformation[orderId] = $"[Id={orderId}] CancelOrder: " + order;
 
                     CheckRateLimiting();
 
@@ -606,9 +602,9 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
 
             var manualResetEvent = new ManualResetEvent(false);
 
-            var requestId = GetNextRequestId();
+            var requestId = GetNextId();
 
-            _requestInformation[requestId] = "GetExecutions: " + symbol;
+            _requestInformation[requestId] = $"[Id={requestId}] GetExecutions: " + symbol;
 
             // define our event handlers
             EventHandler<IB.RequestEndEventArgs> clientOnExecutionDataEnd = (sender, args) =>
@@ -949,7 +945,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
             if (needsNewId)
             {
                 // the order ids are generated for us by the SecurityTransactionManaer
-                var id = GetNextBrokerageOrderId();
+                var id = GetNextId();
                 order.BrokerId.Add(id.ToStringInvariant());
                 ibOrderId = id;
             }
@@ -963,7 +959,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
                 throw new ArgumentException("Expected order with populated BrokerId for updating orders.");
             }
 
-            _requestInformation[ibOrderId] = $"IBPlaceOrder: {order.Symbol.Value} ({contract})";
+            _requestInformation[ibOrderId] = $"[Id={ibOrderId}] IBPlaceOrder: {order.Symbol.Value} ({GetContractDescription(contract)} )";
 
             CheckRateLimiting();
 
@@ -980,7 +976,12 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
 
         private static string GetUniqueKey(Contract contract)
         {
-            return $"{contract} {contract.LastTradeDateOrContractMonth.ToStringInvariant()} {contract.Strike.ToStringInvariant()} {contract.Right}";
+            return $"{contract.ToString().ToUpperInvariant()} {contract.LastTradeDateOrContractMonth.ToStringInvariant()} {contract.Strike.ToStringInvariant()} {contract.Right}";
+        }
+
+        private static string GetContractDescription(Contract contract)
+        {
+            return $"{contract} {contract.PrimaryExch ?? string.Empty} {contract.LastTradeDateOrContractMonth.ToStringInvariant()} {contract.Strike.ToStringInvariant()} {contract.Right}";
         }
 
         private string GetPrimaryExchange(Contract contract, Symbol symbol)
@@ -1041,10 +1042,13 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
         {
             const int timeout = 60; // sec
 
-            ContractDetails details = null;
-            var requestId = GetNextRequestId();
+            var requestId = GetNextId();
 
-            _requestInformation[requestId] = $"GetContractDetails: {symbol.Value} ({contract})";
+            var contractDetailsList = new List<ContractDetails>();
+
+            Log.Trace($"InteractiveBrokersBrokerage.GetContractDetails(): {symbol.Value} ({contract})");
+
+            _requestInformation[requestId] = $"[Id={requestId}] GetContractDetails: {symbol.Value} ({contract})";
 
             var manualResetEvent = new ManualResetEvent(false);
 
@@ -1052,12 +1056,26 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
             EventHandler<IB.ContractDetailsEventArgs> clientOnContractDetails = (sender, args) =>
             {
                 // ignore other requests
-                if (args.RequestId != requestId) return;
-                details = args.ContractDetails;
-                var uniqueKey = GetUniqueKey(contract);
+                if (args.RequestId != requestId)
+                {
+                    return;
+                }
+
+                var details = args.ContractDetails;
+                contractDetailsList.Add(details);
+
+                var uniqueKey = GetUniqueKey(details.Contract);
                 _contractDetails.TryAdd(uniqueKey, details);
-                manualResetEvent.Set();
-                Log.Trace("InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: " + uniqueKey);
+
+                Log.Trace($"InteractiveBrokersBrokerage.GetContractDetails(): clientOnContractDetails event: {uniqueKey}");
+            };
+
+            EventHandler<IB.RequestEndEventArgs> clientOnContractDetailsEnd = (sender, args) =>
+            {
+                if (args.RequestId == requestId)
+                {
+                    manualResetEvent.Set();
+                }
             };
 
             EventHandler<IB.ErrorEventArgs> clientOnError = (sender, args) =>
@@ -1069,6 +1087,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
             };
 
             _client.ContractDetails += clientOnContractDetails;
+            _client.ContractDetailsEnd += clientOnContractDetailsEnd;
             _client.Error += clientOnError;
 
             CheckRateLimiting();
@@ -1083,32 +1102,21 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
 
             // be sure to remove our event handlers
             _client.Error -= clientOnError;
+            _client.ContractDetailsEnd -= clientOnContractDetailsEnd;
             _client.ContractDetails -= clientOnContractDetails;
 
-            return details;
-        }
+            Log.Trace($"InteractiveBrokersBrokerage.GetContractDetails(): contracts found: {contractDetailsList.Count}");
 
-        private string GetFuturesContractExchange(Contract contract, string ticker)
-        {
-            // searching for available contracts on different exchanges
-            var contractDetails = FindContracts(contract, ticker);
-
-            var exchanges = _futuresExchanges.Values.Reverse().ToArray();
-
-            // sorting list of available contracts by exchange priority, taking the top 1
-            return contractDetails
-                    .Select(details => details.Contract.Exchange)
-                    .OrderByDescending(e => Array.IndexOf(exchanges, e))
-                    .FirstOrDefault();
+            return contractDetailsList.FirstOrDefault();
         }
 
         public IEnumerable<ContractDetails> FindContracts(Contract contract, string ticker)
         {
             const int timeout = 60; // sec
 
-            var requestId = GetNextRequestId();
+            var requestId = GetNextId();
 
-            _requestInformation[requestId] = $"FindContracts: {ticker} ({contract})";
+            _requestInformation[requestId] = $"[Id={requestId}] FindContracts: {ticker} ({GetContractDescription(contract)})";
 
             var manualResetEvent = new ManualResetEvent(false);
             var contractDetails = new List<ContractDetails>();
@@ -1853,6 +1861,14 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
                                         _futuresExchanges[symbol.ID.Market] :
                                         symbol.ID.Market;
 
+                var symbolProperties = _symbolPropertiesDatabase.GetSymbolProperties(
+                    symbol.ID.Market,
+                    symbol.ID.Symbol,
+                    SecurityType.Future,
+                    Currencies.USD);
+
+                contract.Multiplier = Convert.ToInt32(symbolProperties.ContractMultiplier).ToStringInvariant();
+
                 contract.IncludeExpired = includeExpired;
             }
 
@@ -2107,7 +2123,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
 
                 default:
                     throw new NotSupportedException(
-                        $"An existing position or open order for an unsupported security type was found: {contract}. " +
+                        $"An existing position or open order for an unsupported security type was found: {GetContractDescription(contract)}. " +
                         "Please manually close the position or cancel the order before restarting the algorithm.");
             }
         }
@@ -2233,10 +2249,10 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
         }
 
         /// <summary>
-        /// Handles the threading issues of creating an IB order ID
+        /// Handles the threading issues of creating an IB OrderId/RequestId/TickerId
         /// </summary>
-        /// <returns>The new IB ID</returns>
-        private int GetNextBrokerageOrderId()
+        /// <returns>The new IB OrderId/RequestId/TickerId</returns>
+        private int GetNextId()
         {
             lock (_nextValidIdLocker)
             {
@@ -2245,16 +2261,6 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
             }
         }
 
-        private int GetNextRequestId()
-        {
-            return Interlocked.Increment(ref _nextRequestId);
-        }
-
-        private int GetNextTickerId()
-        {
-            return Interlocked.Increment(ref _nextTickerId);
-        }
-
         private void HandleBrokerTime(object sender, IB.CurrentTimeUtcEventArgs e)
         {
             // keep track of clock drift
@@ -2322,10 +2328,10 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
                                 return false;
                             }
 
-                            var id = GetNextTickerId();
+                            var id = GetNextId();
                             var contract = CreateContract(subscribeSymbol, false);
 
-                            _requestInformation[id] = $"Subscribe: {symbol.Value} ({contract})";
+                            _requestInformation[id] = $"[Id={id}] Subscribe: {symbol.Value} ({GetContractDescription(contract)})";
 
                             CheckRateLimiting();
 
@@ -2345,7 +2351,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
                             _subscribedSymbols[symbol] = id;
                             _subscribedTickers[id] = new SubscriptionEntry { Symbol = subscribeSymbol };
 
-                            Log.Trace($"InteractiveBrokersBrokerage.Subscribe(): Subscribe Processed: {symbol.Value} ({contract}) # {id}");
+                            Log.Trace($"InteractiveBrokersBrokerage.Subscribe(): Subscribe Processed: {symbol.Value} ({GetContractDescription(contract)}) # {id}");
                             return true;
                         }
                     }
@@ -2739,6 +2745,18 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
             }
             else if (securityType == SecurityType.Future)
             {
+                string market;
+                if (_symbolPropertiesDatabase.TryGetMarket(lookupName, securityType, out market))
+                {
+                    var symbolProperties = _symbolPropertiesDatabase.GetSymbolProperties(
+                        market,
+                        lookupName,
+                        securityType,
+                        Currencies.USD);
+
+                    contract.Multiplier = Convert.ToInt32(symbolProperties.ContractMultiplier).ToStringInvariant();
+                }
+
                 // processing request
                 var results = FindContracts(contract, contract.Symbol);
 
@@ -2833,7 +2851,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
             var startTime = request.Resolution == Resolution.Daily ? request.StartTimeUtc.Date : request.StartTimeUtc;
             var endTime = request.Resolution == Resolution.Daily ? request.EndTimeUtc.Date : request.EndTimeUtc;
 
-            Log.Trace($"InteractiveBrokersBrokerage::GetHistory(): Submitting request: {request.Symbol.Value} ({contract}): {request.Resolution}/{request.TickType} {startTime} UTC -> {endTime} UTC");
+            Log.Trace($"InteractiveBrokersBrokerage::GetHistory(): Submitting request: {request.Symbol.Value} ({GetContractDescription(contract)}): {request.Resolution}/{request.TickType} {startTime} UTC -> {endTime} UTC");
 
             DateTimeZone exchangeTimeZone;
             if (!_symbolExchangeTimeZones.TryGetValue(request.Symbol, out exchangeTimeZone))
@@ -2878,7 +2896,7 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
                 yield return bar;
             }
 
-            Log.Trace($"InteractiveBrokersBrokerage::GetHistory(): Download completed: {request.Symbol.Value} ({contract})");
+            Log.Trace($"InteractiveBrokersBrokerage::GetHistory(): Download completed: {request.Symbol.Value} ({GetContractDescription(contract)})");
         }
 
         private IEnumerable<TradeBar> GetHistory(
@@ -2904,9 +2922,9 @@ namespace QuantConnect.Brokerages.InteractiveBrokers
             {
                 var pacing = false;
                 var historyPiece = new List<TradeBar>();
-                var historicalTicker = GetNextTickerId();
+                var historicalTicker = GetNextId();
 
-                _requestInformation[historicalTicker] = $"GetHistory: {request.Symbol.Value} ({contract})";
+                _requestInformation[historicalTicker] = $"[Id={historicalTicker}] GetHistory: {request.Symbol.Value} ({GetContractDescription(contract)})";
 
                 EventHandler<IB.HistoricalDataEventArgs> clientOnHistoricalData = (sender, args) =>
                 {

Brokerages/QuantConnect.Brokerages.csproj

@@ -350,6 +350,7 @@
     <Compile Include="Bitfinex\BitfinexBrokerageFactory.cs" />
     <Compile Include="Bitfinex\BitfinexBrokerage.Utility.cs" />
     <Compile Include="Bitfinex\BitfinexSubscriptionManager.cs" />
+    <Compile Include="Bitfinex\BitfinexWebSocketChannels.cs" />
     <Compile Include="Bitfinex\BitfinexWebSocketWrapper.cs" />
     <Compile Include="Bitfinex\BitfinexSymbolMapper.cs" />
     <Compile Include="Bitfinex\Messages\Messages.cs" />

Brokerages/Tradier/TradierBrokerage.cs

@@ -124,6 +124,11 @@ namespace QuantConnect.Brokerages.Tradier
             get { return _previousResponseRaw; }
         }
 
+        /// <summary>
+        /// Returns the brokerage account's base currency
+        /// </summary>
+        public override string AccountBaseCurrency => Currencies.USD;
+
         /// <summary>
         /// Create a new Tradier Object:
         /// </summary>

Common/Api/LiveAlgorithmSettings.cs

@@ -28,15 +28,15 @@ namespace QuantConnect.Api
         /// </summary>
         /// <param name="projectId">Id of project from QuantConnect</param>
         /// <param name="compileId">Id of compilation of project from QuantConnect</param>
-        /// <param name="serverType">Server type to run live Algorithm</param>
+        /// <param name="nodeId">Server type to run live Algorithm</param>
         /// <param name="settings"><see cref="BaseLiveAlgorithmSettings ">Live Algorithm Settings</see> for a specific brokerage</param>
         /// <param name="version">The version identifier</param>
-        public LiveAlgorithmApiSettingsWrapper(int projectId, string compileId, string serverType, BaseLiveAlgorithmSettings settings, string version = "-1")
+        public LiveAlgorithmApiSettingsWrapper(int projectId, string compileId, string nodeId, BaseLiveAlgorithmSettings settings, string version = "-1")
         {
             VersionId = version;
             ProjectId = projectId;
             CompileId = compileId;
-            ServerType = serverType;
+            NodeId = nodeId;
             Brokerage = settings;
         }
 
@@ -59,10 +59,10 @@ namespace QuantConnect.Api
         public string CompileId { get; private set; }
 
         /// <summary>
-        /// Type of server being used to run live algorithm
+        /// Id of the node being used to run live algorithm
         /// </summary>
-        [JsonProperty(PropertyName = "serverType")]
-        public string ServerType { get; private set; }
+        [JsonProperty(PropertyName = "nodeId")]
+        public string NodeId { get; private set; }
 
         /// <summary>
         /// The API expects the settings as part of a brokerage object

Common/Brokerages/AlpacaBrokerageModel.cs

@@ -17,7 +17,6 @@ using System.Collections.Generic;
 using System.Linq;
 using QuantConnect.Orders;
 using QuantConnect.Orders.Fees;
-using QuantConnect.Orders.Fills;
 using QuantConnect.Orders.Slippage;
 using QuantConnect.Securities;
 using QuantConnect.Util;
@@ -177,16 +176,6 @@ namespace QuantConnect.Brokerages
             return true;
         }
 
-        /// <summary>
-        /// Gets a new fill model that represents this brokerage's fill behavior
-        /// </summary>
-        /// <param name="security">The security to get fill model for</param>
-        /// <returns>The new fill model for this brokerage</returns>
-        public override IFillModel GetFillModel(Security security)
-        {
-            return new ImmediateFillModel();
-        }
-
         /// <summary>
         /// Gets a new fee model that represents this brokerage's fee structure
         /// </summary>

Common/Brokerages/DefaultBrokerageModel.cs

@@ -187,6 +187,28 @@ namespace QuantConnect.Brokerages
         /// <returns>The new fill model for this brokerage</returns>
         public virtual IFillModel GetFillModel(Security security)
         {
+            switch (security.Type)
+            {
+                case SecurityType.Base:
+                    break;
+                case SecurityType.Equity:
+                    return new EquityFillModel();
+                case SecurityType.Option:
+                    break;
+                case SecurityType.Commodity:
+                    break;
+                case SecurityType.Forex:
+                    break;
+                case SecurityType.Future:
+                    break;
+                case SecurityType.Cfd:
+                    break;
+                case SecurityType.Crypto:
+                    break;
+                default:
+                    throw new ArgumentOutOfRangeException($"{GetType().Name}.GetFillModel: Invalid security type {security.Type}");
+            }
+
             return new ImmediateFillModel();
         }
 

Common/Brokerages/FxcmBrokerageModel.cs

@@ -17,7 +17,6 @@ using System;
 using System.Collections.Generic;
 using QuantConnect.Orders;
 using QuantConnect.Orders.Fees;
-using QuantConnect.Orders.Fills;
 using QuantConnect.Orders.Slippage;
 using QuantConnect.Securities;
 using QuantConnect.Util;
@@ -161,16 +160,6 @@ namespace QuantConnect.Brokerages
             return IsValidOrderPrices(security, order.Type, direction, stopPrice, limitPrice, ref message);
         }
 
-        /// <summary>
-        /// Gets a new fill model that represents this brokerage's fill behavior
-        /// </summary>
-        /// <param name="security">The security to get fill model for</param>
-        /// <returns>The new fill model for this brokerage</returns>
-        public override IFillModel GetFillModel(Security security)
-        {
-            return new ImmediateFillModel();
-        }
-
         /// <summary>
         /// Gets a new fee model that represents this brokerage's fee structure
         /// </summary>

Common/Brokerages/OandaBrokerageModel.cs

@@ -16,7 +16,6 @@
 using System.Collections.Generic;
 using QuantConnect.Orders;
 using QuantConnect.Orders.Fees;
-using QuantConnect.Orders.Fills;
 using QuantConnect.Orders.Slippage;
 using QuantConnect.Securities;
 using QuantConnect.Util;
@@ -104,16 +103,6 @@ namespace QuantConnect.Brokerages
             return true;
         }
 
-        /// <summary>
-        /// Gets a new fill model that represents this brokerage's fill behavior
-        /// </summary>
-        /// <param name="security">The security to get fill model for</param>
-        /// <returns>The new fill model for this brokerage</returns>
-        public override IFillModel GetFillModel(Security security)
-        {
-            return new ImmediateFillModel();
-        }
-
         /// <summary>
         /// Gets a new fee model that represents this brokerage's fee structure
         /// </summary>

Common/Brokerages/TradierBrokerageModel.cs

@@ -17,7 +17,6 @@ using System.Collections.Generic;
 using QuantConnect.Data.Market;
 using QuantConnect.Orders;
 using QuantConnect.Orders.Fees;
-using QuantConnect.Orders.Fills;
 using QuantConnect.Orders.Slippage;
 using QuantConnect.Securities;
 using QuantConnect.Securities.Equity;
@@ -159,16 +158,6 @@ namespace QuantConnect.Brokerages
             }
         }
 
-        /// <summary>
-        /// Gets a new fill model that represents this brokerage's fill behavior
-        /// </summary>
-        /// <param name="security">The security to get fill model for</param>
-        /// <returns>The new fill model for this brokerage</returns>
-        public override IFillModel GetFillModel(Security security)
-        {
-            return new ImmediateFillModel();
-        }
-
         /// <summary>
         /// Gets a new fee model that represents this brokerage's fee structure
         /// </summary>

Common/Data/Auxiliary/FactorFile.cs

@@ -187,9 +187,12 @@ namespace QuantConnect.Data.Auxiliary
         /// <param name="date">The date to check the factor file for a dividend event</param>
         /// <param name="priceFactorRatio">When this function returns true, this value will be populated
         /// with the price factor ratio required to scale the closing value (pf_i/pf_i+1)</param>
-        public bool HasDividendEventOnNextTradingDay(DateTime date, out decimal priceFactorRatio)
+        /// <param name="referencePrice">When this function returns true, this value will be populated
+        /// with the reference raw price, which is the close of the provided date</param>
+        public bool HasDividendEventOnNextTradingDay(DateTime date, out decimal priceFactorRatio, out decimal referencePrice)
         {
             priceFactorRatio = 0;
+            referencePrice = 0;
             var index = SortedFactorFileData.IndexOfKey(date);
             if (index > -1 && index < SortedFactorFileData.Count - 1)
             {
@@ -201,6 +204,7 @@ namespace QuantConnect.Data.Auxiliary
                 if (thisRow.PriceFactor != nextRow.PriceFactor)
                 {
                     priceFactorRatio = thisRow.PriceFactor / nextRow.PriceFactor;
+                    referencePrice = thisRow.ReferencePrice;
                     return true;
                 }
             }
@@ -217,9 +221,15 @@ namespace QuantConnect.Data.Auxiliary
         /// has a split on 1999.03.29, but in the factor file the split factor is applied on
         /// 1999.03.26, which is the first trading day BEFORE the actual split date.
         /// </remarks>
-        public bool HasSplitEventOnNextTradingDay(DateTime date, out decimal splitFactor)
+        /// <param name="date">The date to check the factor file for a split event</param>
+        /// <param name="splitFactor">When this function returns true, this value will be populated
+        /// with the split factor ratio required to scale the closing value</param>
+        /// <param name="referencePrice">When this function returns true, this value will be populated
+        /// with the reference raw price, which is the close of the provided date</param>
+        public bool HasSplitEventOnNextTradingDay(DateTime date, out decimal splitFactor, out decimal referencePrice)
         {
             splitFactor = 1;
+            referencePrice = 0;
             var index = SortedFactorFileData.IndexOfKey(date);
             if (index > -1 && index < SortedFactorFileData.Count - 1)
             {
@@ -231,6 +241,7 @@ namespace QuantConnect.Data.Auxiliary
                 if (thisRow.SplitFactor != nextRow.SplitFactor)
                 {
                     splitFactor = thisRow.SplitFactor / nextRow.SplitFactor;
+                    referencePrice = thisRow.ReferencePrice;
                     return true;
                 }
             }
@@ -264,8 +275,9 @@ namespace QuantConnect.Data.Auxiliary
         /// </summary>
         /// <param name="symbol">The symbol to ues for the dividend and split objects</param>
         /// <param name="exchangeHours">Exchange hours used for resolving the previous trading day</param>
+        /// <param name="decimalPlaces">The number of decimal places to round the dividend's distribution to, defaulting to 2</param>
         /// <returns>All splits and diviends represented by this factor file in chronological order</returns>
-        public List<BaseData> GetSplitsAndDividends(Symbol symbol, SecurityExchangeHours exchangeHours)
+        public List<BaseData> GetSplitsAndDividends(Symbol symbol, SecurityExchangeHours exchangeHours, int decimalPlaces = 2)
         {
             var dividendsAndSplits = new List<BaseData>();
             if (SortedFactorFileData.Count == 0)
@@ -278,7 +290,7 @@ namespace QuantConnect.Data.Auxiliary
             for (var i = SortedFactorFileData.Count - 2; i >= 0; i--)
             {
                 var row = SortedFactorFileData.Values[i];
-                var dividend = row.GetDividend(futureFactorFileRow, symbol, exchangeHours);
+                var dividend = row.GetDividend(futureFactorFileRow, symbol, exchangeHours, decimalPlaces);
                 if (dividend.Distribution != 0m)
                 {
                     dividendsAndSplits.Add(dividend);

Common/Data/Auxiliary/FactorFileRow.cs

@@ -229,8 +229,9 @@ namespace QuantConnect.Data.Auxiliary
         /// <param name="futureFactorFileRow">The next factor file row in time</param>
         /// <param name="symbol">The symbol to use for the dividend</param>
         /// <param name="exchangeHours">Exchange hours used for resolving the previous trading day</param>
+        /// <param name="decimalPlaces">The number of decimal places to round the dividend's distribution to, defaulting to 2</param>
         /// <returns>A new dividend instance</returns>
-        public Dividend GetDividend(FactorFileRow futureFactorFileRow, Symbol symbol, SecurityExchangeHours exchangeHours)
+        public Dividend GetDividend(FactorFileRow futureFactorFileRow, Symbol symbol, SecurityExchangeHours exchangeHours, int decimalPlaces=2)
         {
             if (futureFactorFileRow.PriceFactor == 0m)
             {
@@ -246,7 +247,8 @@ namespace QuantConnect.Data.Auxiliary
                 symbol,
                 previousTradingDay,
                 ReferencePrice,
-                PriceFactor / futureFactorFileRow.PriceFactor
+                PriceFactor / futureFactorFileRow.PriceFactor,
+                decimalPlaces
             );
         }
 
@@ -300,9 +302,9 @@ namespace QuantConnect.Data.Auxiliary
         {
             source = source == null ? "" : $",{source}";
             return $"{Date.ToStringInvariant(DateFormat.EightCharacter)}," +
-                   Invariant($"{Math.Round(PriceFactor, 6).Normalize()},") +
-                   Invariant($"{Math.Round(SplitFactor, 7).Normalize()},") +
-                   Invariant($"{Math.Round(ReferencePrice, 2).Normalize()}") +
+                   Invariant($"{Math.Round(PriceFactor, 7)},") +
+                   Invariant($"{Math.Round(SplitFactor, 8)},") +
+                   Invariant($"{Math.Round(ReferencePrice, 4).Normalize()}") +
                    $"{source}";
         }
 

Common/Data/BaseData.cs

@@ -86,7 +86,6 @@ namespace QuantConnect.Data
         /// The end time of this data. Some data covers spans (trade bars) and as such we want
         /// to know the entire time span covered
         /// </summary>
-        [ProtoMember(3)]
         public virtual DateTime EndTime
         {
             get { return Time; }

Common/Data/Custom/Quiver/Congress.cs

@@ -0,0 +1,35 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using System.Runtime.Serialization;
+
+namespace QuantConnect.Data.Custom.Quiver
+{
+    /// <summary>
+    /// United States of America Legislative Branch House of Congress
+    /// </summary>
+    public enum Congress
+    {
+        /// <summary>
+        /// The United States Senate
+        /// </summary>
+        Senate,
+
+        /// <summary>
+        /// The United States House of Representatives
+        /// </summary>
+        Representatives
+    }
+}

Common/Data/Custom/Quiver/QuiverCongress.cs

@@ -0,0 +1,178 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using Newtonsoft.Json;
+using System;
+using System.IO;
+using NodaTime;
+using ProtoBuf;
+using static QuantConnect.StringExtensions;
+using QuantConnect.Util;
+using Newtonsoft.Json.Converters;
+using QuantConnect.Orders;
+
+namespace QuantConnect.Data.Custom.Quiver
+{
+    /// <summary>
+    /// Personal stock transactions by U.S. Representatives
+    /// </summary>
+    [ProtoContract(SkipConstructor = true)]
+    public class QuiverCongress : BaseData
+    {
+        /// <summary>
+        /// The date the transaction was reported
+        /// </summary>
+        [ProtoMember(10)]
+        [JsonProperty(PropertyName = "ReportDate")]
+        [JsonConverter(typeof(DateTimeJsonConverter), "yyyy-MM-dd")]
+        public DateTime ReportDate { get; set; }
+
+        /// <summary>
+        /// The date the transaction took place
+        /// </summary>
+        [ProtoMember(11)]
+        [JsonProperty(PropertyName = "TransactionDate")]
+        [JsonConverter(typeof(DateTimeJsonConverter), "yyyy-MM-dd")]
+        public DateTime TransactionDate { get; set; }
+
+        /// <summary>
+        /// The Representative making the transaction
+        /// </summary>
+        [ProtoMember(12)]
+        [JsonProperty(PropertyName = "Representative")]
+        public string Representative { get; set; }
+
+        /// <summary>
+        /// The type of transaction
+        /// </summary>
+        [ProtoMember(13)]
+        [JsonProperty(PropertyName = "Transaction")]
+        [JsonConverter(typeof(TransactionDirectionJsonConverter))]
+        public OrderDirection Transaction { get; set; }
+
+        /// <summary>
+        /// The amount of the transaction (in USD)
+        /// </summary>
+        [ProtoMember(14)]
+        [JsonProperty(PropertyName = "Amount")]
+        public decimal? Amount { get; set; }
+
+        /// <summary>
+        /// The House of Congress that the trader belongs to
+        /// </summary>
+        [ProtoMember(15)]
+        [JsonProperty(PropertyName = "House")]
+        [JsonConverter(typeof(StringEnumConverter))]
+        public Congress House { get; set; }
+
+        /// <summary>
+        /// Required for successful Json.NET deserialization
+        /// </summary>
+        public QuiverCongress()
+        {
+        }
+
+        /// <summary>
+        /// Creates a new instance of QuiverCongress from a CSV line
+        /// </summary>
+        /// <param name="csvLine">CSV line</param>
+        public QuiverCongress(string csvLine)
+        {
+            // ReportDate[0], TransactionDate[1], Representative[2], Transaction[3], Amount[4],House[5]
+            var csv = csvLine.Split(',');
+            ReportDate = Parse.DateTimeExact(csv[0], "yyyyMMdd");
+            TransactionDate = Parse.DateTimeExact(csv[1], "yyyyMMdd");
+            Representative = csv[2];
+            var transaction = (TransactionDirection)Enum.Parse(typeof(TransactionDirection), csv[3], true);
+            Transaction = transaction == TransactionDirection.Purchase ? OrderDirection.Buy : OrderDirection.Sell;
+            Amount = csv[4].IfNotNullOrEmpty<decimal?>(s => Parse.Decimal(s));
+            House = (Congress)Enum.Parse(typeof(Congress), csv[5], true);
+            Time = ReportDate;
+        }
+
+        /// <summary>
+        /// Return the Subscription Data Source gained from the URL
+        /// </summary>
+        /// <param name="config">Configuration object</param>
+        /// <param name="date">Date of this source file</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>Subscription Data Source.</returns>
+        public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
+        {
+            if (isLiveMode)
+            {
+                throw new InvalidOperationException($"{nameof(QuiverCongress)} data source is currently not supported in live trading");
+            }
+
+            var source = Path.Combine(
+                Globals.DataFolder,
+                "alternative",
+                "quiver",
+                "congresstrading",
+                $"{config.Symbol.Value.ToLowerInvariant()}.csv"
+            );
+            return new SubscriptionDataSource(source, SubscriptionTransportMedium.LocalFile, FileFormat.Csv);
+        }
+
+        /// <summary>
+        /// Reader converts each line of the data source into BaseData objects.
+        /// </summary>
+        /// <param name="config">Subscription data config setup object</param>
+        /// <param name="line">Content of the source document</param>
+        /// <param name="date">Date of the requested data</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>
+        /// Quiver Congress object
+        /// </returns>
+        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
+        {
+            return new QuiverCongress(line)
+            {
+                Symbol = config.Symbol
+            };
+        }
+
+        /// <summary>
+        /// Formats a string with the Quiver Congress information.
+        /// </summary>
+        public override string ToString()
+        {
+            return Invariant($"{Symbol}({ReportDate}) :: ") +
+                   Invariant($"Transaction Date: {TransactionDate} ") +
+                   Invariant($"Representative: {Representative} ") +
+                   Invariant($"House: {House} ") +
+                   Invariant($"Transaction: {Transaction}") +
+                   Invariant($"Amount: {Amount}");
+        }
+
+        /// <summary>
+        /// Indicates if there is support for mapping
+        /// </summary>
+        /// <returns>True indicates mapping should be used</returns>
+        public override bool RequiresMapping()
+        {
+            return true;
+        }
+
+        /// <summary>
+        /// Specifies the data time zone for this data type. This is useful for custom data types
+        /// </summary>
+        /// <returns>The <see cref="DateTimeZone"/> of this data type</returns>
+        public override DateTimeZone DataTimeZone()
+        {
+            return TimeZones.Utc;
+        }
+    }
+}

Common/Data/Custom/Quiver/QuiverEventsBeta.cs

@@ -0,0 +1,194 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using QuantConnect.Util;
+using System.Collections.Generic;
+using System.Linq;
+using Newtonsoft.Json;
+using System;
+using System.IO;
+using NodaTime;
+using ProtoBuf;
+using static QuantConnect.StringExtensions;
+
+namespace QuantConnect.Data.Custom.Quiver
+{
+    /// <summary>
+    /// Political beta for the specified company
+    /// </summary>
+    [ProtoContract(SkipConstructor = true)]
+    public class QuiverEventsBeta : BaseData
+    {
+
+        /// <summary>
+        /// The date of the events beta calculation
+        /// </summary>
+        [ProtoMember(10)]
+        [JsonProperty(PropertyName = "Date")]
+        [JsonConverter(typeof(DateTimeJsonConverter), "yyyy-MM-dd")]
+        public DateTime Date { get; set; }
+
+        /// <summary>
+        /// Event name (e.g. PresidentialElection2020)
+        /// </summary>
+        [ProtoMember(11)]
+        [JsonProperty(PropertyName = "EventName")]
+        public string EventName { get; set; }
+
+        /// <summary>
+        /// Name for first outcome (e.g. TrumpVictory)
+        /// </summary>
+        [ProtoMember(12)]
+        [JsonProperty(PropertyName = "FirstEventName")]
+        public string FirstEventName { get; set; }
+
+        /// <summary>
+        /// Name for second outcome (e.g. BidenVictory)
+        /// </summary>
+        [ProtoMember(13)]
+        [JsonProperty(PropertyName = "SecondEventName")]
+        public string SecondEventName { get; set; }
+
+        /// <summary>
+        /// Correlation between daily excess returns and daily changes in first event odds
+        /// </summary>
+        [ProtoMember(14)]
+        [JsonProperty(PropertyName = "FirstEventBeta")]
+        public decimal FirstEventBeta { get; set; }
+
+        /// <summary>
+        /// Odds of the first event happening, based on betting markets
+        /// </summary>
+        [ProtoMember(15)]
+        [JsonProperty(PropertyName = "FirstEventOdds")]
+        public decimal FirstEventOdds { get; set; }
+
+        /// <summary>
+        /// Correlation between daily excess returns and daily changes in second event odds
+        /// </summary>
+        [ProtoMember(16)]
+        [JsonProperty(PropertyName = "SecondEventBeta")]
+        public decimal SecondEventBeta { get; set; }
+
+        /// <summary>
+        /// Odds of the second event happening, based on betting markets
+        /// </summary>
+        [ProtoMember(17)]
+        [JsonProperty(PropertyName = "SecondEventOdds")]
+        public decimal SecondEventOdds { get; set; }
+
+        /// <summary>
+        /// Required for successful Json.NET deserialization
+        /// </summary>
+        public QuiverEventsBeta()
+        {
+        }
+
+        /// <summary>
+        /// Creates a new instance of QuiverPoliticalBeta from a CSV line
+        /// </summary>
+        /// <param name="csvLine">CSV line</param>
+        public QuiverEventsBeta(string csvLine)
+        {
+            // Date[0], EventName[1], FirstEventName[2], SecondEventName[3], FirstEventBeta[4], SecondEventBeta[5], FirstEventOdds[6], SecondEventOdds[7]
+            var csv = csvLine.Split(',');
+            Date = Parse.DateTimeExact(csv[0], "yyyyMMdd");
+            EventName = csv[1];
+            FirstEventName = csv[2];
+            SecondEventName = csv[3];
+            FirstEventBeta = csv[4].IfNotNullOrEmpty<decimal>(s => Parse.Decimal(s));
+            SecondEventBeta = csv[5].IfNotNullOrEmpty<decimal>(s => Parse.Decimal(s));
+            FirstEventOdds = csv[6].IfNotNullOrEmpty<decimal>(s => Parse.Decimal(s));
+            SecondEventOdds = csv[7].IfNotNullOrEmpty<decimal>(s => Parse.Decimal(s));
+            Time = Date;
+        }
+
+        /// <summary>
+        /// Return the Subscription Data Source gained from the URL
+        /// </summary>
+        /// <param name="config">Configuration object</param>
+        /// <param name="date">Date of this source file</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>Subscription Data Source.</returns>
+        public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
+        {
+            if (isLiveMode)
+            {
+                throw new InvalidOperationException($"{nameof(QuiverEventsBeta)} data source is currently not supported in live trading");
+            }
+
+            var source = Path.Combine(
+                Globals.DataFolder,
+                "alternative",
+                "quiver",
+                "eventsbeta",
+                $"{config.Symbol.Value.ToLowerInvariant()}.csv"
+            );
+            return new SubscriptionDataSource(source, SubscriptionTransportMedium.LocalFile, FileFormat.Csv);
+        }
+
+        /// <summary>
+        /// Reader converts each line of the data source into BaseData objects.
+        /// </summary>
+        /// <param name="config">Subscription data config setup object</param>
+        /// <param name="line">Content of the source document</param>
+        /// <param name="date">Date of the requested data</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>
+        /// Quiver Political Beta object
+        /// </returns>
+        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
+        {
+
+            return new QuiverEventsBeta(line)
+            {
+                Symbol = config.Symbol
+            };
+        }
+
+        /// <summary>
+        /// Formats a string with the Quiver Events Beta information.
+        /// </summary>
+        public override string ToString()
+        {
+            return Invariant($"{Symbol}({Date}) :: ") +
+                   Invariant($"Event Name: {EventName} ") +
+                   Invariant($"Outcome #1: {FirstEventName}") +
+                   Invariant($"Outcome #2: {SecondEventName}") +
+                   Invariant($"First Outcome Beta: {FirstEventBeta}") +
+                   Invariant($"Second Outcome Beta: {SecondEventBeta}") +
+                   Invariant($"First Outcome Odds: {FirstEventOdds}") +
+                   Invariant($"Second Outcome Odds: {SecondEventOdds}");
+        }
+
+        /// <summary>
+        /// Indicates if there is support for mapping
+        /// </summary>
+        /// <returns>True indicates mapping should be used</returns>
+        public override bool RequiresMapping()
+        {
+            return true;
+        }
+
+        /// <summary>
+        /// Specifies the data time zone for this data type. This is useful for custom data types
+        /// </summary>
+        /// <returns>The <see cref="DateTimeZone"/> of this data type</returns>
+        public override DateTimeZone DataTimeZone()
+        {
+            return TimeZones.Utc;
+        }
+    }
+}

Common/Data/Custom/Quiver/QuiverWallStreetBets.cs

@@ -0,0 +1,136 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using QuantConnect.Util;
+using Newtonsoft.Json;
+using System;
+using System.IO;
+using NodaTime;
+using ProtoBuf;
+using static QuantConnect.StringExtensions;
+
+namespace QuantConnect.Data.Custom.Quiver
+{
+    /// <summary>
+    /// Mentions of the given company's ticker in the WallStreetBets daily discussion thread
+    /// </summary>
+    [ProtoContract(SkipConstructor = true)]
+    public class QuiverWallStreetBets : BaseData
+    {
+        /// <summary>
+        /// Date of the daily discussion thread
+        /// </summary>
+        [ProtoMember(10)]
+        [JsonProperty(PropertyName = "Date")]
+        [JsonConverter(typeof(DateTimeJsonConverter), "yyyy-MM-dd")]
+        public DateTime Date { get; set; }
+
+        /// <summary>
+        /// The number of mentions on the given date
+        /// </summary>
+        [ProtoMember(11)]
+        [JsonProperty(PropertyName = "Count")]
+        public int Mentions { get; set; }
+
+        /// <summary>
+        /// Required for successful Json.NET deserialization
+        /// </summary>
+        public QuiverWallStreetBets()
+        {
+        }
+
+        /// <summary>
+        /// Creates a new instance of QuiverWallStreetBets from a CSV line
+        /// </summary>
+        /// <param name="csvLine">CSV line</param>
+        public QuiverWallStreetBets(string csvLine)
+        {
+            // Date[0], Mentions[1]
+            var csv = csvLine.Split(',');
+            Date = Parse.DateTimeExact(csv[0], "yyyyMMdd");
+            Mentions = Parse.Int(csv[1]);
+            Time = Date;
+        }
+
+        /// <summary>
+        /// Return the Subscription Data Source gained from the URL
+        /// </summary>
+        /// <param name="config">Configuration object</param>
+        /// <param name="date">Date of this source file</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>Subscription Data Source.</returns>
+        public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
+        {
+            if (isLiveMode)
+            {
+                throw new InvalidOperationException($"{nameof(QuiverWallStreetBets)} data source is currently not supported in live trading");
+            }
+
+            var source = Path.Combine(
+                Globals.DataFolder,
+                "alternative",
+                "quiver",
+                "wallstreetbets",
+                $"{config.Symbol.Value.ToLowerInvariant()}.csv"
+            );
+            return new SubscriptionDataSource(source, SubscriptionTransportMedium.LocalFile, FileFormat.Csv);
+        }
+
+        /// <summary>
+        /// Reader converts each line of the data source into BaseData objects.
+        /// </summary>
+        /// <param name="config">Subscription data config setup object</param>
+        /// <param name="line">Content of the source document</param>
+        /// <param name="date">Date of the requested data</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>
+        /// Quiver WallStreetBets object
+        /// </returns>
+        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
+        {
+            return new QuiverWallStreetBets(line)
+            {
+                Symbol = config.Symbol
+            };
+        }
+
+        /// <summary>
+        /// Formats a string with the Quiver WallStreetBets information.
+        /// </summary>
+        public override string ToString()
+        {
+            return Invariant($"{Symbol}({Date}) :: ") +
+                   Invariant($"WallStreetBets Mentions: {Mentions} ");
+        }
+
+        /// <summary>
+        /// Indicates if there is support for mapping
+        /// </summary>
+        /// <returns>True indicates mapping should be used</returns>
+        public override bool RequiresMapping()
+        {
+            return true;
+        }
+
+        /// <summary>
+        /// Specifies the data time zone for this data type. This is useful for custom data types
+        /// </summary>
+        /// <returns>The <see cref="DateTimeZone"/> of this data type</returns>
+        public override DateTimeZone DataTimeZone()
+        {
+            return TimeZones.Utc;
+        }
+    }
+}

Common/Data/Custom/Quiver/QuiverWikipedia.cs

@@ -0,0 +1,156 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using QuantConnect.Util;
+using Newtonsoft.Json;
+using System;
+using System.IO;
+using NodaTime;
+using ProtoBuf;
+using static QuantConnect.StringExtensions;
+
+namespace QuantConnect.Data.Custom.Quiver
+{
+    /// <summary>
+    /// Wikipedia Page Views for the specified company
+    /// </summary>
+    [ProtoContract(SkipConstructor = true)]
+    public class QuiverWikipedia : BaseData
+    {
+        /// <summary>
+        /// The date of the Page View count
+        /// </summary>
+        [ProtoMember(10)]
+        [JsonProperty(PropertyName = "Date")]
+        [JsonConverter(typeof(DateTimeJsonConverter), "yyyy-MM-dd")]
+        public DateTime Date { get; set; }
+
+        /// <summary>
+        /// The company's Wikipedia Page Views on the given date
+        /// </summary>
+        [ProtoMember(11)]
+        [JsonProperty(PropertyName = "Views")]
+        public decimal? PageViews { get; set; }
+
+        /// <summary>
+        /// The view count % change over the week prior to the date.
+        /// Represented as a whole number (e.g. 100% = 100.0)
+        /// </summary>
+        [ProtoMember(12)]
+        [JsonProperty(PropertyName = "pct_change_week")]
+        public decimal? WeekPercentChange { get; set; }
+
+        /// <summary>
+        /// The view count % change over the month prior to the date
+        /// Represented as a whole number (e.g. 100% = 100.0)
+        /// </summary>
+        [ProtoMember(13)]
+        [JsonProperty(PropertyName = "pct_change_month")]
+        public decimal? MonthPercentChange { get; set; }
+
+        /// <summary>
+        /// Required for successful Json.NET deserialization
+        /// </summary>
+        public QuiverWikipedia()
+        {
+        }
+
+        /// <summary>
+        /// Creates a new instance of QuiverWikipedia from a CSV line
+        /// </summary>
+        /// <param name="csvLine">CSV line</param>
+        public QuiverWikipedia(string csvLine)
+        {
+            // Date[0], Ticker[1], PageViews[2], WeekPercentChange[3], MonthPercentChange[4]
+            var csv = csvLine.Split(',');
+            Date = Parse.DateTimeExact(csv[0], "yyyyMMdd");
+            PageViews = csv[1].IfNotNullOrEmpty<decimal?>(s => Parse.Decimal(s));
+            WeekPercentChange = csv[2].IfNotNullOrEmpty<decimal?>(s => Parse.Decimal(s));
+            MonthPercentChange = csv[3].IfNotNullOrEmpty<decimal?>(s => Parse.Decimal(s));
+            Time = Date;
+        }
+
+        /// <summary>
+        /// Return the Subscription Data Source gained from the URL
+        /// </summary>
+        /// <param name="config">Configuration object</param>
+        /// <param name="date">Date of this source file</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>Subscription Data Source.</returns>
+        public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
+        {
+            if (isLiveMode)
+            {
+                throw new InvalidOperationException($"{nameof(QuiverWikipedia)} data source is currently not supported in live trading");
+            }
+
+            var source = Path.Combine(
+                Globals.DataFolder,
+                "alternative",
+                "quiver",
+                "wikipedia",
+                $"{config.Symbol.Value.ToLowerInvariant()}.csv"
+            );
+            return new SubscriptionDataSource(source, SubscriptionTransportMedium.LocalFile, FileFormat.Csv);
+        }
+
+        /// <summary>
+        /// Reader converts each line of the data source into BaseData objects.
+        /// </summary>
+        /// <param name="config">Subscription data config setup object</param>
+        /// <param name="line">Content of the source document</param>
+        /// <param name="date">Date of the requested data</param>
+        /// <param name="isLiveMode">true if we're in live mode, false for backtesting mode</param>
+        /// <returns>
+        /// Quiver Wikipedia object
+        /// </returns>
+        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
+        {
+            return new QuiverWikipedia(line)
+            {
+                Symbol = config.Symbol
+            };
+        }
+
+        /// <summary>
+        /// Formats a string with the Quiver Wikipedia information.
+        /// </summary>
+        public override string ToString()
+        {
+            return Invariant($"{Symbol}({Date}) :: ") +
+                   Invariant($"Followers: {PageViews} ") +
+                   Invariant($"% Change Week: {WeekPercentChange}") +
+                   Invariant($"% Change Month: {MonthPercentChange}");
+        }
+
+        /// <summary>
+        /// Indicates if there is support for mapping
+        /// </summary>
+        /// <returns>True indicates mapping should be used</returns>
+        public override bool RequiresMapping()
+        {
+            return true;
+        }
+
+        /// <summary>
+        /// Specifies the data time zone for this data type. This is useful for custom data types
+        /// </summary>
+        /// <returns>The <see cref="DateTimeZone"/> of this data type</returns>
+        public override DateTimeZone DataTimeZone()
+        {
+            return TimeZones.Utc;
+        }
+    }
+}

Common/Data/Custom/Quiver/TransactionDirection.cs

@@ -0,0 +1,36 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using QuantConnect.Orders;
+
+namespace QuantConnect.Data.Custom.Quiver
+{
+    /// <summary>
+    /// Transaction direction
+    /// </summary>
+    /// <remarks>We use this enum to successfully deserialize responses from the API</remarks>
+    public enum TransactionDirection
+    {
+        /// <summary>
+        /// Buy, equivalent to <see cref="OrderDirection.Buy"/>
+        /// </summary>
+        Purchase,
+
+        /// <summary>
+        /// Sell, equivalent to <see cref="OrderDirection.Sell"/>
+        /// </summary>
+        Sale
+    };
+}

Common/Data/Custom/Quiver/TransactionDirectionJsonConverter.cs

@@ -0,0 +1,36 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using QuantConnect.Orders;
+using QuantConnect.Util;
+
+namespace QuantConnect.Data.Custom.Quiver
+{
+    /// <summary>
+    /// Converts Quiver Quantitative <see cref="TransactionDirection"/> to <see cref="OrderDirection"/>
+    /// </summary>
+    public class TransactionDirectionJsonConverter : TypeChangeJsonConverter<OrderDirection, string>
+    {
+        protected override string Convert(OrderDirection value)
+        {
+            return value == OrderDirection.Buy ? "purchase" : "sale";
+        }
+
+        protected override OrderDirection Convert(string value)
+        {
+            return value.ToLowerInvariant() == "purchase" ? OrderDirection.Buy : OrderDirection.Sell;
+        }
+    }
+}

Common/Data/Custom/Tiingo/TiingoNewsJsonConverter.cs

@@ -1,4 +1,4 @@
-/*
+/*
  * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
  * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
  *
@@ -18,6 +18,7 @@ using System;
 using System.Collections.Generic;
 using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
+using QuantConnect.Logging;
 
 namespace QuantConnect.Data.Custom.Tiingo
 {
@@ -118,8 +119,14 @@ namespace QuantConnect.Data.Custom.Tiingo
             var symbols = new List<Symbol>();
             foreach (var tiingoTicker in tickers)
             {
-                var ticker = TiingoSymbolMapper.GetLeanTicker(tiingoTicker.ToString());
-
+                var rawTicker = tiingoTicker.ToString();
+                if (rawTicker.Contains(" ") || rawTicker.Contains("|"))
+                {
+                    Log.Trace($"TiingoNewsJsonConverter.DeserializeNews(): Article ID {articleID}, ignoring ticker [{rawTicker}] because it contains space or pipe character.");
+                    continue;
+                }
+                
+                var ticker = TiingoSymbolMapper.GetLeanTicker(rawTicker);
                 var sid = SecurityIdentifier.GenerateEquity(
                     ticker,
                     // for now we suppose USA market

Common/Data/Market/Dividend.cs

@@ -78,9 +78,10 @@ namespace QuantConnect.Data.Market
         /// <param name="date">The date</param>
         /// <param name="referencePrice">The previous day's closing price</param>
         /// <param name="priceFactorRatio">The ratio of the price factors, pf_i/pf_i+1</param>
-        public static Dividend Create(Symbol symbol, DateTime date, decimal referencePrice, decimal priceFactorRatio)
+        /// <param name="decimalPlaces">The number of decimal places to round the dividend's distribution to, defaulting to 2</param>
+        public static Dividend Create(Symbol symbol, DateTime date, decimal referencePrice, decimal priceFactorRatio, int decimalPlaces = 2)
         {
-            var distribution = ComputeDistribution(referencePrice, priceFactorRatio);
+            var distribution = ComputeDistribution(referencePrice, priceFactorRatio, decimalPlaces);
             return new Dividend(symbol, date, distribution, referencePrice);
         }
 
@@ -91,7 +92,7 @@ namespace QuantConnect.Data.Market
         /// <param name="priceFactorRatio">Price factor ratio pf_i/pf_i+1</param>
         /// <param name="decimalPlaces">The number of decimal places to round the result to, defaulting to 2</param>
         /// <returns>The distribution rounded to the specified number of decimal places, defaulting to 2</returns>
-        public static decimal ComputeDistribution(decimal close, decimal priceFactorRatio, int decimalPlaces = 2)
+        public static decimal ComputeDistribution(decimal close, decimal priceFactorRatio, int decimalPlaces)
         {
             return Math.Round(close - close * priceFactorRatio, decimalPlaces);
         }

Common/Data/Market/Tick.cs

@@ -17,6 +17,7 @@ using System;
 using System.Globalization;
 using System.IO;
 using System.Runtime.CompilerServices;
+using Newtonsoft.Json;
 using ProtoBuf;
 using QuantConnect.Logging;
 using QuantConnect.Util;
@@ -30,6 +31,8 @@ namespace QuantConnect.Data.Market
     [ProtoContract(SkipConstructor = true)]
     public class Tick : BaseData
     {
+        private uint? _parsedSaleCondition;
+
         /// <summary>
         /// Type of the Tick: Trade or Quote.
         /// </summary>
@@ -51,9 +54,28 @@ namespace QuantConnect.Data.Market
         /// <summary>
         /// Sale condition for the tick.
         /// </summary>
-        [ProtoMember(13)]
         public string SaleCondition = "";
 
+        /// <summary>
+        /// For performance parsed sale condition for the tick.
+        /// </summary>
+        [JsonIgnore]
+        public uint ParsedSaleCondition
+        {
+            get
+            {
+                if (!_parsedSaleCondition.HasValue)
+                {
+                    _parsedSaleCondition = uint.Parse(SaleCondition, NumberStyles.HexNumber, CultureInfo.InvariantCulture);
+                }
+                return _parsedSaleCondition.Value;
+            }
+            set
+            {
+                _parsedSaleCondition = value;
+            }
+        }
+
         /// <summary>
         /// Bool whether this is a suspicious tick
         /// </summary>

Common/DataProviderEvents.cs

@@ -180,6 +180,11 @@ namespace QuantConnect
         /// </summary>
         public BaseData LastBaseData { get; }
 
+        /// <summary>
+        /// The last raw security price we have
+        /// </summary>
+        public decimal? LastRawPrice { get; }
+
         /// <summary>
         /// Initializes a new instance of the <see cref="NewTradableDateEventArgs"/> class
         /// </summary>
@@ -187,11 +192,13 @@ namespace QuantConnect
         /// <param name="lastBaseData">The last <see cref="BaseData"/> of the
         /// <see cref="Security"/> for which we are enumerating</param>
         /// <param name="symbol">The <see cref="Symbol"/> of the new tradable date</param>
-        public NewTradableDateEventArgs(DateTime date, BaseData lastBaseData, Symbol symbol)
+        /// <param name="lastRawPrice">The last raw security price we have</param>
+        public NewTradableDateEventArgs(DateTime date, BaseData lastBaseData, Symbol symbol, decimal? lastRawPrice)
             : base(symbol)
         {
             Date = date;
             LastBaseData = lastBaseData;
+            LastRawPrice = lastRawPrice;
         }
     }
 }

Common/Extensions.cs

@@ -47,7 +47,7 @@ using QuantConnect.Securities;
 using QuantConnect.Util;
 using Timer = System.Timers.Timer;
 using static QuantConnect.StringExtensions;
-using System.Runtime.CompilerServices;
+using Microsoft.IO;
 using QuantConnect.Data.Auxiliary;
 using QuantConnect.Securities.Option;
 
@@ -58,9 +58,52 @@ namespace QuantConnect
     /// </summary>
     public static class Extensions
     {
+        private static RecyclableMemoryStreamManager MemoryManager = new RecyclableMemoryStreamManager();
+        private static ConcurrentBag<Guid> Guids = new ConcurrentBag<Guid>();
+
         private static readonly Dictionary<IntPtr, PythonActivator> PythonActivators
             = new Dictionary<IntPtr, PythonActivator>();
 
+        /// <summary>
+        /// Will return a memory stream using the <see cref="RecyclableMemoryStreamManager"/> instance.
+        /// </summary>
+        /// <remarks>For performance will reuse a memory stream guid per thread. So</remarks>
+        /// <returns></returns>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static MemoryStream GetMemoryStream(Guid guid)
+        {
+            return MemoryManager.GetStream(guid);
+        }
+
+        /// <summary>
+        /// Gets a unique id. Should be returned using <see cref="ReturnId"/>
+        /// </summary>
+        /// <remarks>Creating a new <see cref="Guid"/> is expensive</remarks>
+        /// <remarks>Used for <see cref="GetMemoryStream"/></remarks>
+        /// <returns>A unused <see cref="Guid"/></returns>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static Guid RentId()
+        {
+            Guid guid;
+            if (!Guids.TryTake(out guid))
+            {
+                guid = new Guid();
+            }
+            return guid;
+        }
+
+        /// <summary>
+        /// Returns a rented unique id <see cref="RentId"/>
+        /// </summary>
+        /// <remarks>Creating a new <see cref="Guid"/> is expensive</remarks>
+        /// <remarks>Used for <see cref="GetMemoryStream"/></remarks>
+        /// <param name="guid">The guid to return</param>
+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        public static void ReturnId(Guid guid)
+        {
+            Guids.Add(guid);
+        }
+
         /// <summary>
         /// Serialize a list of ticks using protobuf
         /// </summary>
@@ -68,11 +111,16 @@ namespace QuantConnect
         /// <returns>The resulting byte array</returns>
         public static byte[] ProtobufSerialize(this List<Tick> ticks)
         {
-            using (var stream = new MemoryStream())
+            var guid = RentId();
+            byte[] result;
+            using (var stream = GetMemoryStream(guid))
             {
                 Serializer.Serialize(stream, ticks);
-                return stream.ToArray();
+                result = stream.ToArray();
             }
+
+            ReturnId(guid);
+            return result;
         }
 
         /// <summary>
@@ -82,7 +130,10 @@ namespace QuantConnect
         /// <returns>The resulting byte array</returns>
         public static byte[] ProtobufSerialize(this IBaseData baseData)
         {
-            using (var stream = new MemoryStream())
+            var guid = RentId();
+
+            byte[] result;
+            using (var stream = GetMemoryStream(guid))
             {
                 switch (baseData.DataType)
                 {
@@ -99,8 +150,11 @@ namespace QuantConnect
                         Serializer.SerializeWithLengthPrefix(stream, baseData as BaseData, PrefixStyle.Base128, 1);
                         break;
                 }
-                return stream.ToArray();
+                result = stream.ToArray();
             }
+            ReturnId(guid);
+
+            return result;
         }
 
         /// <summary>

Common/Interfaces/IApi.cs

@@ -262,13 +262,6 @@ namespace QuantConnect.Interfaces
         /// <param name="message">Message for the algorithm status event</param>
         void SetAlgorithmStatus(string algorithmId, AlgorithmStatus status, string message = "");
 
-        /// <summary>
-        /// Will get the prices for requested symbols
-        /// </summary>
-        /// <param name="symbols">Symbols for which the price is requested</param>
-        /// <returns><see cref="Prices"/></returns>
-        PricesList ReadPrices(IEnumerable<Symbol> symbols);
-
         /// <summary>
         /// Send the statistics to storage for performance tracking.
         /// </summary>
@@ -292,22 +285,6 @@ namespace QuantConnect.Interfaces
         /// <param name="body">The email message body</param>
         void SendUserEmail(string algorithmId, string subject, string body);
 
-        /// <summary>
-        /// Gets all split events between the specified times. From and to are inclusive.
-        /// </summary>
-        /// <param name="from">The first date to get splits for</param>
-        /// <param name="to">The last date to get splits for</param>
-        /// <returns>A list of all splits in the specified range</returns>
-        List<Data.Market.Split> GetSplits(DateTime from, DateTime to);
-
-        /// <summary>
-        /// Gets all dividend events between the specified times. From and to are inclusive.
-        /// </summary>
-        /// <param name="from">The first date to get dividend for</param>
-        /// <param name="to">The last date to get dividend for</param>
-        /// <returns>A list of all dividend in the specified range</returns>
-        List<Data.Market.Dividend> GetDividends(DateTime from, DateTime to);
-
         /// <summary>
         /// Local implementation for downloading data to algorithms
         /// </summary>

Common/Orders/Fills/EquityFillModel.cs

@@ -0,0 +1,587 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using System;
+using System.Linq;
+using QuantConnect.Data;
+using QuantConnect.Data.Market;
+using QuantConnect.Python;
+using QuantConnect.Orders.Fees;
+using QuantConnect.Securities;
+
+namespace QuantConnect.Orders.Fills
+{
+    /// <summary>
+    /// Provides a base class for all fill models
+    /// </summary>
+    public class EquityFillModel : IFillModel
+    {
+        /// <summary>
+        /// The parameters instance to be used by the different XxxxFill() implementations
+        /// </summary>
+        protected FillModelParameters Parameters { get; set; }
+
+        /// <summary>
+        /// This is required due to a limitation in PythonNet to resolved overriden methods.
+        /// When Python calls a C# method that calls a method that's overriden in python it won't
+        /// run the python implementation unless the call is performed through python too.
+        /// </summary>
+        protected FillModelPythonWrapper PythonWrapper;
+
+        /// <summary>
+        /// Used to set the <see cref="FillModelPythonWrapper"/> instance if any
+        /// </summary>
+        public void SetPythonWrapper(FillModelPythonWrapper pythonWrapper)
+        {
+            PythonWrapper = pythonWrapper;
+        }
+
+        /// <summary>
+        /// Return an order event with the fill details
+        /// </summary>
+        /// <param name="parameters">A <see cref="FillModelParameters"/> object containing the security and order</param>
+        /// <returns>Order fill information detailing the average price and quantity filled.</returns>
+        public virtual Fill Fill(FillModelParameters parameters)
+        {
+            // Important: setting the parameters is required because it is
+            // consumed by the different XxxxFill() implementations
+            Parameters = parameters;
+
+            var order = parameters.Order;
+            OrderEvent orderEvent;
+            switch (order.Type)
+            {
+                case OrderType.Market:
+                    orderEvent = PythonWrapper != null
+                        ? PythonWrapper.MarketFill(parameters.Security, parameters.Order as MarketOrder)
+                        : MarketFill(parameters.Security, parameters.Order as MarketOrder);
+                    break;
+                case OrderType.Limit:
+                    orderEvent = PythonWrapper != null
+                        ? PythonWrapper.LimitFill(parameters.Security, parameters.Order as LimitOrder)
+                        : LimitFill(parameters.Security, parameters.Order as LimitOrder);
+                    break;
+                case OrderType.StopMarket:
+                    orderEvent = PythonWrapper != null
+                        ? PythonWrapper.StopMarketFill(parameters.Security, parameters.Order as StopMarketOrder)
+                        : StopMarketFill(parameters.Security, parameters.Order as StopMarketOrder);
+                    break;
+                case OrderType.StopLimit:
+                    orderEvent = PythonWrapper != null
+                        ? PythonWrapper.StopLimitFill(parameters.Security, parameters.Order as StopLimitOrder)
+                        : StopLimitFill(parameters.Security, parameters.Order as StopLimitOrder);
+                    break;
+                case OrderType.MarketOnOpen:
+                    orderEvent = PythonWrapper != null
+                        ? PythonWrapper.MarketOnOpenFill(parameters.Security, parameters.Order as MarketOnOpenOrder)
+                        : MarketOnOpenFill(parameters.Security, parameters.Order as MarketOnOpenOrder);
+                    break;
+                case OrderType.MarketOnClose:
+                    orderEvent = PythonWrapper != null
+                        ? PythonWrapper.MarketOnCloseFill(parameters.Security, parameters.Order as MarketOnCloseOrder)
+                        : MarketOnCloseFill(parameters.Security, parameters.Order as MarketOnCloseOrder);
+                    break;
+                default:
+                    throw new ArgumentOutOfRangeException();
+            }
+            return new Fill(orderEvent);
+        }
+
+        /// <summary>
+        /// Default market fill model for the base security class. Fills at the last traded price.
+        /// </summary>
+        /// <param name="asset">Security asset we're filling</param>
+        /// <param name="order">Order packet to model</param>
+        /// <returns>Order fill information detailing the average price and quantity filled.</returns>
+        public virtual OrderEvent MarketFill(Security asset, MarketOrder order)
+        {
+            //Default order event to return.
+            var utcTime = asset.LocalTime.ConvertToUtc(asset.Exchange.TimeZone);
+            var fill = new OrderEvent(order, utcTime, OrderFee.Zero);
+
+            if (order.Status == OrderStatus.Canceled) return fill;
+
+            // make sure the exchange is open/normal market hours before filling
+            if (!IsExchangeOpen(asset, false)) return fill;
+
+            var prices = GetPricesCheckingPythonWrapper(asset, order.Direction);
+            var pricesEndTimeUtc = prices.EndTime.ConvertToUtc(asset.Exchange.TimeZone);
+
+            // if the order is filled on stale (fill-forward) data, set a warning message on the order event
+            if (pricesEndTimeUtc.Add(Parameters.StalePriceTimeSpan) < order.Time)
+            {
+                fill.Message = $"Warning: fill at stale price ({prices.EndTime.ToStringInvariant()} {asset.Exchange.TimeZone})";
+            }
+
+            //Order [fill]price for a market order model is the current security price
+            fill.FillPrice = prices.Current;
+            fill.Status = OrderStatus.Filled;
+
+            //Calculate the model slippage: e.g. 0.01c
+            var slip = asset.SlippageModel.GetSlippageApproximation(asset, order);
+
+            //Apply slippage
+            switch (order.Direction)
+            {
+                case OrderDirection.Buy:
+                    fill.FillPrice += slip;
+                    break;
+                case OrderDirection.Sell:
+                    fill.FillPrice -= slip;
+                    break;
+            }
+
+            // assume the order completely filled
+            fill.FillQuantity = order.Quantity;
+
+            return fill;
+        }
+
+        /// <summary>
+        /// Default stop fill model implementation in base class security. (Stop Market Order Type)
+        /// </summary>
+        /// <param name="asset">Security asset we're filling</param>
+        /// <param name="order">Order packet to model</param>
+        /// <returns>Order fill information detailing the average price and quantity filled.</returns>
+        /// <seealso cref="MarketFill(Security, MarketOrder)"/>
+        public virtual OrderEvent StopMarketFill(Security asset, StopMarketOrder order)
+        {
+            //Default order event to return.
+            var utcTime = asset.LocalTime.ConvertToUtc(asset.Exchange.TimeZone);
+            var fill = new OrderEvent(order, utcTime, OrderFee.Zero);
+
+            //If its cancelled don't need anymore checks:
+            if (order.Status == OrderStatus.Canceled) return fill;
+
+            // make sure the exchange is open/normal market hours before filling
+            if (!IsExchangeOpen(asset, false)) return fill;
+
+            //Get the range of prices in the last bar:
+            var prices = GetPricesCheckingPythonWrapper(asset, order.Direction);
+            var pricesEndTime = prices.EndTime.ConvertToUtc(asset.Exchange.TimeZone);
+
+            // do not fill on stale data
+            if (pricesEndTime <= order.Time) return fill;
+
+            //Calculate the model slippage: e.g. 0.01c
+            var slip = asset.SlippageModel.GetSlippageApproximation(asset, order);
+
+            //Check if the Stop Order was filled: opposite to a limit order
+            switch (order.Direction)
+            {
+                case OrderDirection.Sell:
+                    //-> 1.1 Sell Stop: If Price below setpoint, Sell:
+                    if (prices.Low < order.StopPrice)
+                    {
+                        fill.Status = OrderStatus.Filled;
+                        // Assuming worse case scenario fill - fill at lowest of the stop & asset price.
+                        fill.FillPrice = Math.Min(order.StopPrice, prices.Current - slip);
+                        // assume the order completely filled
+                        fill.FillQuantity = order.Quantity;
+                    }
+                    break;
+
+                case OrderDirection.Buy:
+                    //-> 1.2 Buy Stop: If Price Above Setpoint, Buy:
+                    if (prices.High > order.StopPrice)
+                    {
+                        fill.Status = OrderStatus.Filled;
+                        // Assuming worse case scenario fill - fill at highest of the stop & asset price.
+                        fill.FillPrice = Math.Max(order.StopPrice, prices.Current + slip);
+                        // assume the order completely filled
+                        fill.FillQuantity = order.Quantity;
+                    }
+                    break;
+            }
+
+            return fill;
+        }
+
+        /// <summary>
+        /// Default stop limit fill model implementation in base class security. (Stop Limit Order Type)
+        /// </summary>
+        /// <param name="asset">Security asset we're filling</param>
+        /// <param name="order">Order packet to model</param>
+        /// <returns>Order fill information detailing the average price and quantity filled.</returns>
+        /// <seealso cref="StopMarketFill(Security, StopMarketOrder)"/>
+        /// <remarks>
+        ///     There is no good way to model limit orders with OHLC because we never know whether the market has
+        ///     gapped past our fill price. We have to make the assumption of a fluid, high volume market.
+        ///
+        ///     Stop limit orders we also can't be sure of the order of the H - L values for the limit fill. The assumption
+        ///     was made the limit fill will be done with closing price of the bar after the stop has been triggered..
+        /// </remarks>
+        public virtual OrderEvent StopLimitFill(Security asset, StopLimitOrder order)
+        {
+            //Default order event to return.
+            var utcTime = asset.LocalTime.ConvertToUtc(asset.Exchange.TimeZone);
+            var fill = new OrderEvent(order, utcTime, OrderFee.Zero);
+
+            //If its cancelled don't need anymore checks:
+            if (order.Status == OrderStatus.Canceled) return fill;
+
+            // make sure the exchange is open before filling -- allow pre/post market fills to occur
+            if (!IsExchangeOpen(
+                asset,
+                Parameters.ConfigProvider
+                    .GetSubscriptionDataConfigs(asset.Symbol)
+                    .IsExtendedMarketHours()))
+            {
+                return fill;
+            }
+
+            //Get the range of prices in the last bar:
+            var prices = GetPricesCheckingPythonWrapper(asset, order.Direction);
+            var pricesEndTime = prices.EndTime.ConvertToUtc(asset.Exchange.TimeZone);
+
+            // do not fill on stale data
+            if (pricesEndTime <= order.Time) return fill;
+
+            //Check if the Stop Order was filled: opposite to a limit order
+            switch (order.Direction)
+            {
+                case OrderDirection.Buy:
+                    //-> 1.2 Buy Stop: If Price Above Setpoint, Buy:
+                    if (prices.High > order.StopPrice || order.StopTriggered)
+                    {
+                        order.StopTriggered = true;
+
+                        // Fill the limit order, using closing price of bar:
+                        // Note > Can't use minimum price, because no way to be sure minimum wasn't before the stop triggered.
+                        if (prices.Current < order.LimitPrice)
+                        {
+                            fill.Status = OrderStatus.Filled;
+                            fill.FillPrice = Math.Min(prices.High, order.LimitPrice);
+                            // assume the order completely filled
+                            fill.FillQuantity = order.Quantity;
+                        }
+                    }
+                    break;
+
+                case OrderDirection.Sell:
+                    //-> 1.1 Sell Stop: If Price below setpoint, Sell:
+                    if (prices.Low < order.StopPrice || order.StopTriggered)
+                    {
+                        order.StopTriggered = true;
+
+                        // Fill the limit order, using minimum price of the bar
+                        // Note > Can't use minimum price, because no way to be sure minimum wasn't before the stop triggered.
+                        if (prices.Current > order.LimitPrice)
+                        {
+                            fill.Status = OrderStatus.Filled;
+                            fill.FillPrice = Math.Max(prices.Low, order.LimitPrice);
+                            // assume the order completely filled
+                            fill.FillQuantity = order.Quantity;
+                        }
+                    }
+                    break;
+            }
+
+            return fill;
+        }
+
+        /// <summary>
+        /// Default limit order fill model in the base security class.
+        /// </summary>
+        /// <param name="asset">Security asset we're filling</param>
+        /// <param name="order">Order packet to model</param>
+        /// <returns>Order fill information detailing the average price and quantity filled.</returns>
+        /// <seealso cref="StopMarketFill(Security, StopMarketOrder)"/>
+        /// <seealso cref="MarketFill(Security, MarketOrder)"/>
+        public virtual OrderEvent LimitFill(Security asset, LimitOrder order)
+        {
+            //Initialise;
+            var utcTime = asset.LocalTime.ConvertToUtc(asset.Exchange.TimeZone);
+            var fill = new OrderEvent(order, utcTime, OrderFee.Zero);
+
+            //If its cancelled don't need anymore checks:
+            if (order.Status == OrderStatus.Canceled) return fill;
+
+            // make sure the exchange is open before filling -- allow pre/post market fills to occur
+            if (!IsExchangeOpen(asset,
+                Parameters.ConfigProvider
+                    .GetSubscriptionDataConfigs(asset.Symbol)
+                    .IsExtendedMarketHours()))
+            {
+                return fill;
+            }
+            //Get the range of prices in the last bar:
+            var prices = GetPricesCheckingPythonWrapper(asset, order.Direction);
+            var pricesEndTime = prices.EndTime.ConvertToUtc(asset.Exchange.TimeZone);
+
+            // do not fill on stale data
+            if (pricesEndTime <= order.Time) return fill;
+
+            //-> Valid Live/Model Order:
+            switch (order.Direction)
+            {
+                case OrderDirection.Buy:
+                    //Buy limit seeks lowest price
+                    if (prices.Low < order.LimitPrice)
+                    {
+                        //Set order fill:
+                        fill.Status = OrderStatus.Filled;
+                        // fill at the worse price this bar or the limit price, this allows far out of the money limits
+                        // to be executed properly
+                        fill.FillPrice = Math.Min(prices.High, order.LimitPrice);
+                        // assume the order completely filled
+                        fill.FillQuantity = order.Quantity;
+                    }
+                    break;
+                case OrderDirection.Sell:
+                    //Sell limit seeks highest price possible
+                    if (prices.High > order.LimitPrice)
+                    {
+                        fill.Status = OrderStatus.Filled;
+                        // fill at the worse price this bar or the limit price, this allows far out of the money limits
+                        // to be executed properly
+                        fill.FillPrice = Math.Max(prices.Low, order.LimitPrice);
+                        // assume the order completely filled
+                        fill.FillQuantity = order.Quantity;
+                    }
+                    break;
+            }
+
+            return fill;
+        }
+
+        /// <summary>
+        /// Market on Open Fill Model. Return an order event with the fill details
+        /// </summary>
+        /// <param name="asset">Asset we're trading with this order</param>
+        /// <param name="order">Order to be filled</param>
+        /// <returns>Order fill information detailing the average price and quantity filled.</returns>
+        public virtual OrderEvent MarketOnOpenFill(Security asset, MarketOnOpenOrder order)
+        {
+            var utcTime = asset.LocalTime.ConvertToUtc(asset.Exchange.TimeZone);
+            var fill = new OrderEvent(order, utcTime, OrderFee.Zero);
+
+            if (order.Status == OrderStatus.Canceled) return fill;
+
+            // MOO should never fill on the same bar or on stale data
+            // Imagine the case where we have a thinly traded equity, ASUR, and another liquid
+            // equity, say SPY, SPY gets data every minute but ASUR, if not on fill forward, maybe
+            // have large gaps, in which case the currentBar.EndTime will be in the past
+            // ASUR  | | |      [order]        | | | | | | |
+            //  SPY  | | | | | | | | | | | | | | | | | | | |
+            var currentBar = asset.GetLastData();
+            var localOrderTime = order.Time.ConvertFromUtc(asset.Exchange.TimeZone);
+            if (currentBar == null || localOrderTime >= currentBar.EndTime) return fill;
+
+            // if the MOO was submitted during market the previous day, wait for a day to turn over
+            if (asset.Exchange.DateTimeIsOpen(localOrderTime) && localOrderTime.Date == asset.LocalTime.Date)
+            {
+                return fill;
+            }
+
+            // wait until market open
+            // make sure the exchange is open/normal market hours before filling
+            if (!IsExchangeOpen(asset, false)) return fill;
+
+            fill.FillPrice = GetPricesCheckingPythonWrapper(asset, order.Direction).Open;
+            fill.Status = OrderStatus.Filled;
+            //Calculate the model slippage: e.g. 0.01c
+            var slip = asset.SlippageModel.GetSlippageApproximation(asset, order);
+
+            //Apply slippage
+            switch (order.Direction)
+            {
+                case OrderDirection.Buy:
+                    fill.FillPrice += slip;
+                    // assume the order completely filled
+                    fill.FillQuantity = order.Quantity;
+                    break;
+                case OrderDirection.Sell:
+                    fill.FillPrice -= slip;
+                    // assume the order completely filled
+                    fill.FillQuantity = order.Quantity;
+                    break;
+            }
+
+            return fill;
+        }
+
+        /// <summary>
+        /// Market on Close Fill Model. Return an order event with the fill details
+        /// </summary>
+        /// <param name="asset">Asset we're trading with this order</param>
+        /// <param name="order">Order to be filled</param>
+        /// <returns>Order fill information detailing the average price and quantity filled.</returns>
+        public virtual OrderEvent MarketOnCloseFill(Security asset, MarketOnCloseOrder order)
+        {
+            var utcTime = asset.LocalTime.ConvertToUtc(asset.Exchange.TimeZone);
+            var fill = new OrderEvent(order, utcTime, OrderFee.Zero);
+
+            if (order.Status == OrderStatus.Canceled) return fill;
+
+            var localOrderTime = order.Time.ConvertFromUtc(asset.Exchange.TimeZone);
+            var nextMarketClose = asset.Exchange.Hours.GetNextMarketClose(localOrderTime, false);
+
+            // wait until market closes after the order time
+            if (asset.LocalTime < nextMarketClose)
+            {
+                return fill;
+            }
+            // make sure the exchange is open/normal market hours before filling
+            if (!IsExchangeOpen(asset, false)) return fill;
+
+            fill.FillPrice = GetPricesCheckingPythonWrapper(asset, order.Direction).Close;
+            fill.Status = OrderStatus.Filled;
+            //Calculate the model slippage: e.g. 0.01c
+            var slip = asset.SlippageModel.GetSlippageApproximation(asset, order);
+
+            //Apply slippage
+            switch (order.Direction)
+            {
+                case OrderDirection.Buy:
+                    fill.FillPrice += slip;
+                    // assume the order completely filled
+                    fill.FillQuantity = order.Quantity;
+                    break;
+                case OrderDirection.Sell:
+                    fill.FillPrice -= slip;
+                    // assume the order completely filled
+                    fill.FillQuantity = order.Quantity;
+                    break;
+            }
+
+            return fill;
+        }
+
+        /// <summary>
+        /// This is required due to a limitation in PythonNet to resolved
+        /// overriden methods. <see cref="GetPrices"/>
+        /// </summary>
+        private Prices GetPricesCheckingPythonWrapper(Security asset, OrderDirection direction)
+        {
+            if (PythonWrapper != null)
+            {
+                var prices = PythonWrapper.GetPricesInternal(asset, direction);
+                return new Prices(prices.EndTime, prices.Current, prices.Open, prices.High, prices.Low, prices.Close);
+            }
+            return GetPrices(asset, direction);
+        }
+
+        /// <summary>
+        /// Get the minimum and maximum price for this security in the last bar:
+        /// </summary>
+        /// <param name="asset">Security asset we're checking</param>
+        /// <param name="direction">The order direction, decides whether to pick bid or ask</param>
+        protected virtual Prices GetPrices(Security asset, OrderDirection direction)
+        {
+            var low = asset.Low;
+            var high = asset.High;
+            var open = asset.Open;
+            var close = asset.Close;
+            var current = asset.Price;
+            var endTime = asset.Cache.GetData()?.EndTime ?? DateTime.MinValue;
+
+            if (direction == OrderDirection.Hold)
+            {
+                return new Prices(endTime, current, open, high, low, close);
+            }
+
+            // Only fill with data types we are subscribed to
+            var subscriptionTypes = Parameters.ConfigProvider
+                .GetSubscriptionDataConfigs(asset.Symbol)
+                .Select(x => x.Type).ToList();
+            // Tick
+            var tick = asset.Cache.GetData<Tick>();
+            if (subscriptionTypes.Contains(typeof(Tick)) && tick != null)
+            {
+                var price = direction == OrderDirection.Sell ? tick.BidPrice : tick.AskPrice;
+                if (price != 0m)
+                {
+                    return new Prices(tick.EndTime, price, 0, 0, 0, 0);
+                }
+
+                // If the ask/bid spreads are not available for ticks, try the price
+                price = tick.Price;
+                if (price != 0m)
+                {
+                    return new Prices(tick.EndTime, price, 0, 0, 0, 0);
+                }
+            }
+
+            // Quote
+            var quoteBar = asset.Cache.GetData<QuoteBar>();
+            if (subscriptionTypes.Contains(typeof(QuoteBar)) && quoteBar != null)
+            {
+                var bar = direction == OrderDirection.Sell ? quoteBar.Bid : quoteBar.Ask;
+                if (bar != null)
+                {
+                    return new Prices(quoteBar.EndTime, bar);
+                }
+            }
+
+            // Trade
+            var tradeBar = asset.Cache.GetData<TradeBar>();
+            if (subscriptionTypes.Contains(typeof(TradeBar)) && tradeBar != null)
+            {
+                return new Prices(tradeBar);
+            }
+
+            return new Prices(endTime, current, open, high, low, close);
+        }
+
+        /// <summary>
+        /// Determines if the exchange is open using the current time of the asset
+        /// </summary>
+        protected static bool IsExchangeOpen(Security asset, bool isExtendedMarketHours)
+        {
+            if (!asset.Exchange.DateTimeIsOpen(asset.LocalTime))
+            {
+                // if we're not open at the current time exactly, check the bar size, this handle large sized bars (hours/days)
+                var currentBar = asset.GetLastData();
+                if (asset.LocalTime.Date != currentBar.EndTime.Date
+                    || !asset.Exchange.IsOpenDuringBar(currentBar.Time, currentBar.EndTime, isExtendedMarketHours))
+                {
+                    return false;
+                }
+            }
+            return true;
+        }
+
+        public class Prices
+        {
+            public readonly DateTime EndTime;
+            public readonly decimal Current;
+            public readonly decimal Open;
+            public readonly decimal High;
+            public readonly decimal Low;
+            public readonly decimal Close;
+
+            public Prices(IBaseDataBar bar)
+                : this(bar.EndTime, bar.Close, bar.Open, bar.High, bar.Low, bar.Close)
+            {
+            }
+
+            public Prices(DateTime endTime, IBar bar)
+                : this(endTime, bar.Close, bar.Open, bar.High, bar.Low, bar.Close)
+            {
+            }
+
+            public Prices(DateTime endTime, decimal current, decimal open, decimal high, decimal low, decimal close)
+            {
+                EndTime = endTime;
+                Current = current;
+                Open = open == 0 ? current : open;
+                High = high == 0 ? current : high;
+                Low = low == 0 ? current : low;
+                Close = close == 0 ? current : close;
+            }
+        }
+    }
+}

Common/Orders/Fills/FillModel.cs

@@ -498,7 +498,7 @@ namespace QuantConnect.Orders.Fills
                 .Select(x => x.Type).ToList();
             // Tick
             var tick = asset.Cache.GetData<Tick>();
-            if (subscriptionTypes.Contains(typeof(Tick)) && tick != null)
+            if (tick != null && subscriptionTypes.Contains(typeof(Tick)))
             {
                 var price = direction == OrderDirection.Sell ? tick.BidPrice : tick.AskPrice;
                 if (price != 0m)
@@ -516,7 +516,7 @@ namespace QuantConnect.Orders.Fills
 
             // Quote
             var quoteBar = asset.Cache.GetData<QuoteBar>();
-            if (subscriptionTypes.Contains(typeof(QuoteBar)) && quoteBar != null)
+            if (quoteBar != null && subscriptionTypes.Contains(typeof(QuoteBar)))
             {
                 var bar = direction == OrderDirection.Sell ? quoteBar.Bid : quoteBar.Ask;
                 if (bar != null)
@@ -527,7 +527,7 @@ namespace QuantConnect.Orders.Fills
 
             // Trade
             var tradeBar = asset.Cache.GetData<TradeBar>();
-            if (subscriptionTypes.Contains(typeof(TradeBar)) && tradeBar != null)
+            if (tradeBar != null && subscriptionTypes.Contains(typeof(TradeBar)))
             {
                 return new Prices(tradeBar);
             }

Common/Python/FillModelPythonWrapper.cs

@@ -150,5 +150,16 @@ namespace QuantConnect.Python
                 return (_model.GetPrices(asset, direction) as PyObject).GetAndDispose<Prices>();
             }
         }
+
+        /// <summary>
+        /// Get the minimum and maximum price for this security in the last bar:
+        /// </summary>
+        /// <param name="asset">Security asset we're checking</param>
+        /// <param name="direction">The order direction, decides whether to pick bid or ask</param>
+        /// <remarks>This method was implemented temporarily to help the refactoring of fill models (GH #4567)</remarks>
+        internal Prices GetPricesInternal(Security asset, OrderDirection direction)
+        {
+            return GetPrices(asset, direction);
+        }
     }
 }

Common/QuantConnect.csproj

@@ -121,6 +121,9 @@
     <Reference Include="MathNet.Numerics, Version=3.19.0.0, Culture=neutral, processorArchitecture=MSIL">
       <HintPath>..\packages\MathNet.Numerics.3.19.0\lib\net40\MathNet.Numerics.dll</HintPath>
     </Reference>
+    <Reference Include="Microsoft.IO.RecyclableMemoryStream, Version=1.3.5.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\packages\Microsoft.IO.RecyclableMemoryStream.1.3.5\lib\net46\Microsoft.IO.RecyclableMemoryStream.dll</HintPath>
+    </Reference>
     <Reference Include="Newtonsoft.Json, Version=10.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL">
       <HintPath>..\packages\Newtonsoft.Json.10.0.3\lib\net45\Newtonsoft.Json.dll</HintPath>
     </Reference>
@@ -267,6 +270,13 @@
     <Compile Include="Chart.cs" />
     <Compile Include="ChartPoint.cs" />
     <Compile Include="Data\Channel.cs" />
+    <Compile Include="Data\Custom\Quiver\Congress.cs" />
+    <Compile Include="Data\Custom\Quiver\QuiverCongress.cs" />
+    <Compile Include="Data\Custom\Quiver\QuiverEventsBeta.cs" />
+    <Compile Include="Data\Custom\Quiver\QuiverWallStreetBets.cs" />
+    <Compile Include="Data\Custom\Quiver\QuiverWikipedia.cs" />
+    <Compile Include="Data\Custom\Quiver\TransactionDirection.cs" />
+    <Compile Include="Data\Custom\Quiver\TransactionDirectionJsonConverter.cs" />
     <Compile Include="Data\EventBasedDataQueueHandlerSubscriptionManager.cs" />
     <Compile Include="Data\IDataAggregator.cs" />
     <Compile Include="Data\Auxiliary\MappingExtensions.cs" />
@@ -372,6 +382,7 @@
     <Compile Include="Orders\Fees\AlphaStreamsFeeModel.cs" />
     <Compile Include="Orders\Fees\ModifiedFillQuantityOrderFee.cs" />
     <Compile Include="Orders\Fills\Fill.cs" />
+    <Compile Include="Orders\Fills\EquityFillModel.cs" />
     <Compile Include="Orders\Fills\FillModelParameters.cs" />
     <Compile Include="Orders\Fees\FeeModel.cs" />
     <Compile Include="Orders\Fees\OrderFee.cs" />
@@ -399,6 +410,8 @@
     <Compile Include="Securities\BuyingPowerModel.cs" />
     <Compile Include="Securities\BuyingPowerModelExtensions.cs" />
     <Compile Include="Securities\BuyingPowerParameters.cs" />
+    <Compile Include="Securities\ContractSecurityFilterUniverse.cs" />
+    <Compile Include="Securities\Future\FutureSymbol.cs" />
     <Compile Include="Securities\GetMaximumOrderQuantityForDeltaBuyingPowerParameters.cs" />
     <Compile Include="Securities\RegisteredSecurityDataTypesProvider.cs" />
     <Compile Include="Securities\ErrorCurrencyConverter.cs" />

Common/Scheduling/DateRules.cs

@@ -16,6 +16,7 @@
 
 using System;
 using System.Collections.Generic;
+using System.Globalization;
 using System.Linq;
 using NodaTime;
 using QuantConnect.Securities;
@@ -86,14 +87,14 @@ namespace QuantConnect.Scheduling
         /// <summary>
         /// Specifies an event should fire on each of the specified days of week
         /// </summary>
-        /// <param name="day">The day the event shouls fire</param>
+        /// <param name="day">The day the event should fire</param>
         /// <returns>A date rule that fires on every specified day of week</returns>
         public IDateRule Every(DayOfWeek day) => Every(new[] { day });
 
         /// <summary>
         /// Specifies an event should fire on each of the specified days of week
         /// </summary>
-        /// <param name="days">The days the event shouls fire</param>
+        /// <param name="days">The days the event should fire</param>
         /// <returns>A date rule that fires on every specified day of week</returns>
         public IDateRule Every(params DayOfWeek[] days)
         {
@@ -113,96 +114,154 @@ namespace QuantConnect.Scheduling
         /// <summary>
         /// Specifies an event should fire every day the symbol is trading
         /// </summary>
-        /// <param name="symbol">The symbol whose exchange is used to determine tradeable dates</param>
+        /// <param name="symbol">The symbol whose exchange is used to determine tradable dates</param>
         /// <returns>A date rule that fires every day the specified symbol trades</returns>
         public IDateRule EveryDay(Symbol symbol)
         {
-            var security = GetSecurity(symbol);
-            return new FuncDateRule($"{symbol.Value}: EveryDay", (start, end) => Time.EachTradeableDay(security, start, end));
+            var securitySchedule = GetSecuritySchedule(symbol);
+            return new FuncDateRule($"{symbol.Value}: EveryDay", (start, end) => Time.EachTradeableDay(securitySchedule, start, end));
         }
 
         /// <summary>
-        /// Specifies an event should fire on the first of each month
+        /// Specifies an event should fire on the first of each month + offset
         /// </summary>
-        /// <returns>A date rule that fires on the first of each month</returns>
-        public IDateRule MonthStart()
+        /// <param name="daysOffset"> The amount of days to offset the schedule by; must be between 0 and 30.</param>
+        /// <returns>A date rule that fires on the first of each month + offset</returns>
+        public IDateRule MonthStart(int daysOffset = 0)
         {
-            return new FuncDateRule("MonthStart", (start, end) => MonthStartIterator(null, start, end));
+            return new FuncDateRule(GetName(null, "MonthStart", daysOffset), (start, end) => MonthIterator(null, start, end, daysOffset, true));
         }
 
         /// <summary>
-        /// Specifies an event should fire on the first tradeable date for the specified
-        /// symbol of each month
+        /// Specifies an event should fire on the first tradable date + offset for the specified symbol of each month
         /// </summary>
-        /// <param name="symbol">The symbol whose exchange is used to determine the first
-        /// tradeable date of the month</param>
-        /// <returns>A date rule that fires on the first tradeable date for the specified security each month</returns>
-        public IDateRule MonthStart(Symbol symbol)
+        /// <param name="symbol">The symbol whose exchange is used to determine the first tradable date of the month</param>
+        /// <param name="daysOffset"> The amount of tradable days to offset the schedule by; must be between 0 and 30</param>
+        /// <returns>A date rule that fires on the first tradable date + offset for the
+        /// specified security each month</returns>
+        public IDateRule MonthStart(Symbol symbol, int daysOffset = 0)
         {
-            return new FuncDateRule($"{symbol.Value}: MonthStart", (start, end) => MonthStartIterator(GetSecurity(symbol), start, end));
+            // Check that our offset is allowed
+            if (daysOffset < 0 || 30 < daysOffset)
+            {
+                throw new ArgumentOutOfRangeException(nameof(daysOffset), "DateRules.MonthStart() : Offset must be between 0 and 30");
+            }
+
+            // Create the new DateRule and return it
+            return new FuncDateRule(GetName(symbol, "MonthStart", daysOffset), (start, end) => MonthIterator(GetSecuritySchedule(symbol), start, end, daysOffset, true));
         }
 
         /// <summary>
         /// Specifies an event should fire on the last of each month
         /// </summary>
-        /// <returns>A date rule that fires on the last of each month</returns>
-        public IDateRule MonthEnd()
+        /// <param name="daysOffset"> The amount of days to offset the schedule by; must be between 0 and 30</param>
+        /// <returns>A date rule that fires on the last of each month - offset</returns>
+        public IDateRule MonthEnd(int daysOffset = 0)
         {
-            return new FuncDateRule("MonthEnd", (start, end) => MonthEndIterator(null, start, end));
+            return new FuncDateRule(GetName(null, "MonthEnd", -daysOffset), (start, end) => MonthIterator(null, start, end, daysOffset, false));
         }
 
         /// <summary>
-        /// Specifies an event should fire on the last tradeable date for the specified
-        /// symbol of each month
+        /// Specifies an event should fire on the last tradable date - offset for the specified symbol of each month
         /// </summary>
-        /// <param name="symbol">The symbol whose exchange is used to determine the last
-        /// tradeable date of the month</param>
-        /// <returns>A date rule that fires on the last tradeable date for the specified security each month</returns>
-        public IDateRule MonthEnd(Symbol symbol)
+        /// <param name="symbol">The symbol whose exchange is used to determine the last tradable date of the month</param>
+        /// <param name="daysOffset">The amount of tradable days to offset the schedule by; must be between 0 and 30.</param>
+        /// <returns>A date rule that fires on the last tradable date - offset for the specified security each month</returns>
+        public IDateRule MonthEnd(Symbol symbol, int daysOffset = 0)
         {
-            return new FuncDateRule($"{symbol.Value}: MonthEnd", (start, end) => MonthEndIterator(GetSecurity(symbol), start, end));
+            // Check that our offset is allowed
+            if (daysOffset < 0 || 30 < daysOffset)
+            {
+                throw new ArgumentOutOfRangeException(nameof(daysOffset), "DateRules.MonthEnd() : Offset must be between 0 and 30");
+            }
+
+            // Create the new DateRule and return it
+            return new FuncDateRule(GetName(symbol, "MonthEnd", -daysOffset), (start, end) => MonthIterator(GetSecuritySchedule(symbol), start, end, daysOffset, false));
         }
 
         /// <summary>
-        /// Specifies an event should fire on Monday each week
+        /// Specifies an event should fire on Monday + offset each week
         /// </summary>
-        /// <returns>A date rule that fires on Monday each week</returns>
-        public IDateRule WeekStart()
+        /// <param name="daysOffset">The amount of days to offset monday by; must be between 0 and 6</param>
+        /// <returns>A date rule that fires on Monday + offset each week</returns>
+        public IDateRule WeekStart(int daysOffset = 0)
         {
-            return new FuncDateRule("WeekStart", (start, end) => WeekStartIterator(null, start, end));
+            // Check that our offset is allowed
+            if (daysOffset < 0 || 6 < daysOffset)
+            {
+                throw new ArgumentOutOfRangeException(nameof(daysOffset), "DateRules.WeekStart() : Offset must be between 0 and 6");
+            }
+
+            return new FuncDateRule(GetName(null, "WeekStart", daysOffset), (start, end) => WeekIterator(null, start, end, daysOffset, true));
         }
 
         /// <summary>
-        /// Specifies an event should fire on the first tradeable date for the specified
-        /// symbol of each week
+        /// Specifies an event should fire on the first tradable date + offset for the specified
+        /// symbol each week
         /// </summary>
         /// <param name="symbol">The symbol whose exchange is used to determine the first
         /// tradeable date of the week</param>
-        /// <returns>A date rule that fires on the first tradeable date for the specified security each week</returns>
-        public IDateRule WeekStart(Symbol symbol)
+        /// <param name="daysOffset">The amount of tradable days to offset the first tradable day by</param>
+        /// <returns>A date rule that fires on the first + offset tradable date for the specified
+        /// security each week</returns>
+        public IDateRule WeekStart(Symbol symbol, int daysOffset = 0)
         {
-            return new FuncDateRule($"{symbol.Value}: WeekStart", (start, end) => WeekStartIterator(GetSecurity(symbol), start, end));
+            var securitySchedule = GetSecuritySchedule(symbol);
+            var tradingDays = securitySchedule.MarketHours.Values
+                .Where(x => x.IsClosedAllDay == false).OrderBy(x => x.DayOfWeek).ToList();
+
+            // Limit offsets to securities weekly schedule
+            if (daysOffset > tradingDays.Count - 1)
+            {
+                throw new ArgumentOutOfRangeException(nameof(daysOffset),
+                    $"DateRules.WeekStart() : {tradingDays.First().DayOfWeek}+{daysOffset} is out of range for {symbol}'s schedule," +
+                    $" please use an offset between 0 - {tradingDays.Count - 1}; Schedule : {string.Join(", ", tradingDays.Select(x => x.DayOfWeek))}");
+            }
+
+            // Create the new DateRule and return it
+            return new FuncDateRule(GetName(symbol, "WeekStart", daysOffset), (start, end) => WeekIterator(securitySchedule, start, end, daysOffset, true));
         }
 
         /// <summary>
-        /// Specifies an event should fire on Friday each week
+        /// Specifies an event should fire on Friday - offset
         /// </summary>
+        /// <param name="daysOffset"> The amount of days to offset Friday by; must be between 0 and 6 </param>
         /// <returns>A date rule that fires on Friday each week</returns>
-        public IDateRule WeekEnd()
+        public IDateRule WeekEnd(int daysOffset = 0)
         {
-            return new FuncDateRule("WeekEnd", (start, end) => WeekEndIterator(null, start, end));
+            // Check that our offset is allowed
+            if (daysOffset < 0 || 6 < daysOffset)
+            {
+                throw new ArgumentOutOfRangeException("daysOffset", "DateRules.WeekEnd() : Offset must be between 0 and 6");
+            }
+
+            return new FuncDateRule(GetName(null, "WeekEnd", -daysOffset), (start, end) => WeekIterator(null, start, end, daysOffset, false));
         }
 
         /// <summary>
-        /// Specifies an event should fire on the last tradeable date for the specified
+        /// Specifies an event should fire on the last - offset tradable date for the specified
         /// symbol of each week
         /// </summary>
         /// <param name="symbol">The symbol whose exchange is used to determine the last
-        /// tradeable date of the week</param>
-        /// <returns>A date rule that fires on the last tradeable date for the specified security each week</returns>
-        public IDateRule WeekEnd(Symbol symbol)
+        /// tradable date of the week</param>
+        /// <param name="daysOffset"> The amount of tradable days to offset the last tradable day by each week</param>
+        /// <returns>A date rule that fires on the last - offset tradable date for the specified security each week</returns>
+        public IDateRule WeekEnd(Symbol symbol, int daysOffset = 0)
         {
-            return new FuncDateRule($"{symbol.Value}: WeekEnd", (start, end) => WeekEndIterator(GetSecurity(symbol), start, end));
+            var securitySchedule = GetSecuritySchedule(symbol);
+            var tradingDays = securitySchedule.MarketHours.Values
+                .Where(x => x.IsClosedAllDay == false).OrderBy(x => x.DayOfWeek).ToList();
+
+            // Limit offsets to securities weekly schedule
+            if (daysOffset > tradingDays.Count - 1)
+            {
+                throw new ArgumentOutOfRangeException(nameof(daysOffset),
+                    $"DateRules.WeekEnd() : {tradingDays.Last().DayOfWeek}-{daysOffset} is out of range for {symbol}'s schedule," +
+                    $" please use an offset between 0 - {tradingDays.Count - 1}; Schedule : {string.Join(", ", tradingDays.Select(x => x.DayOfWeek))}");
+            }
+
+            // Create the new DateRule and return it
+            return new FuncDateRule(GetName(symbol, "WeekEnd", -daysOffset), (start, end) => WeekIterator(securitySchedule, start, end, daysOffset, false));
         }
 
         /// <summary>
@@ -210,126 +269,151 @@ namespace QuantConnect.Scheduling
         /// </summary>
         /// <param name="symbol">The security's symbol to search for</param>
         /// <returns>The security object matching the given symbol</returns>
-        private Security GetSecurity(Symbol symbol)
+        private SecurityExchangeHours GetSecuritySchedule(Symbol symbol)
         {
             Security security;
             if (!_securities.TryGetValue(symbol, out security))
             {
                 throw new KeyNotFoundException(symbol.Value + " not found in portfolio. Request this data when initializing the algorithm.");
             }
-            return security;
+            return security.Exchange.Hours;
         }
 
-        private static IEnumerable<DateTime> MonthStartIterator(Security security, DateTime start, DateTime end)
+        /// <summary>
+        /// Determine the string representation for a given rule 
+        /// </summary>
+        /// <param name="symbol">Symbol for the rule</param>
+        /// <param name="ruleType">Rule type in string form</param>
+        /// <param name="offset">The amount of offset on this rule</param>
+        /// <returns></returns>
+        private static string GetName(Symbol symbol, string ruleType, int offset)
         {
-            if (security == null)
-            {
-                foreach (var date in Time.EachDay(start, end))
-                {
-                    // fire on the first of each month
-                    if (date.Day == 1) yield return date;
-                }
-                yield break;
-            }
+            // Convert our offset to +#, -#, or empty string if 0
+            var offsetString = offset.ToString("+#;-#;''", CultureInfo.InvariantCulture);
+            var name = symbol == null ? $"{ruleType}{offsetString}" : $"{symbol.Value}: {ruleType}{offsetString}";
 
-            // start a month back so we can properly resolve the first event (we may have passed it)
-            var aMonthBeforeStart = start.AddMonths(-1);
-            int lastMonth = aMonthBeforeStart.Month;
-            foreach (var date in Time.EachTradeableDay(security, aMonthBeforeStart, end))
-            {
-                if (date.Month != lastMonth)
-                {
-                    if (date >= start)
-                    {
-                        // only emit if the date is on or after the start
-                        // the date may be before here because we backed up a month
-                        // to properly resolve the first tradeable date
-                        yield return date;
-                    }
-                    lastMonth = date.Month;
-                }
-            }
+            return name;
         }
 
-        private static IEnumerable<DateTime> MonthEndIterator(Security security, DateTime start, DateTime end)
+
+        /// <summary>
+        /// Get the closest trading day to a given DateTime for a given <see cref="SecurityExchangeHours"/>.
+        /// </summary>
+        /// <param name="securityExchangeHours"><see cref="SecurityExchangeHours"/> object with schedule for this Security</param>
+        /// <param name="baseDay">The day to base our search from</param>
+        /// <param name="offset">Amount to offset the schedule by tradable days</param>
+        /// <param name="searchForward">Search into the future for the closest day if true; into the past if false</param>
+        /// <param name="boundary">The boundary DateTime on the resulting day</param>
+        /// <returns></returns>
+        private static DateTime GetScheduledDay(SecurityExchangeHours securityExchangeHours, DateTime baseDay, int offset, bool searchForward, DateTime? boundary = null)
         {
-            foreach (var date in Time.EachDay(start, end))
+            // By default the scheduled date is the given day
+            var scheduledDate = baseDay;
+
+            // If its not open on this day find the next trading day by searching in the given direction
+            if (!securityExchangeHours.IsDateOpen(scheduledDate))
             {
-                if (date.Day == DateTime.DaysInMonth(date.Year, date.Month))
+                scheduledDate = searchForward
+                    ? securityExchangeHours.GetNextTradingDay(scheduledDate)
+                    : securityExchangeHours.GetPreviousTradingDay(scheduledDate);
+            }
+
+            // Offset the scheduled day accordingly
+            for (var i = 0; i < offset; i++)
+            {
+                scheduledDate = searchForward
+                    ? securityExchangeHours.GetNextTradingDay(scheduledDate)
+                    : securityExchangeHours.GetPreviousTradingDay(scheduledDate);
+            }
+
+            // If there is a boundary ensure we enforce it
+            if (boundary.HasValue)
+            {
+                // If we are searching forward and the resulting date is after this boundary we
+                // revert to the last tradable day equal to or less than boundary
+                if (searchForward && scheduledDate > boundary)
                 {
-                    if (security == null)
-                    {
-                        // fire on the last of each month
-                        yield return date;
-                    }
-                    else
-                    {
-                        // find previous date when market is open
-                        var currentDate = date;
-                        while (!security.Exchange.Hours.IsDateOpen(currentDate))
-                        {
-                            currentDate = currentDate.AddDays(-1);
-                        }
-                        yield return currentDate;
-                    }
+                    scheduledDate = GetScheduledDay(securityExchangeHours, (DateTime)boundary, 0, false);
+                }
+
+                // If we are searching backward and the resulting date is after this boundary we
+                // revert to the last tradable day equal to or greater than boundary
+                if (!searchForward && scheduledDate < boundary)
+                {
+                    scheduledDate = GetScheduledDay(securityExchangeHours, (DateTime)boundary, 0, true);
                 }
             }
+
+            return scheduledDate;
         }
 
-        private static IEnumerable<DateTime> WeekStartIterator(Security security, DateTime start, DateTime end)
+        private static IEnumerable<DateTime> MonthIterator(SecurityExchangeHours securitySchedule, DateTime start, DateTime end, int offset, bool searchForward)
         {
-            var skippedMarketClosedDay = false;
+            // No schedule means no security, set to open everyday
+            if (securitySchedule == null)
+            {
+                securitySchedule = SecurityExchangeHours.AlwaysOpen(TimeZones.NewYork);
+            }
 
             foreach (var date in Time.EachDay(start, end))
             {
-                if (security == null)
+                var daysInMonth = DateTime.DaysInMonth(date.Year, date.Month);
+
+                // Searching forward the first of the month is baseDay, with boundary being the last
+                // Searching backward the last of the month is baseDay, with boundary being the first
+                var baseDate = searchForward? new DateTime(date.Year, date.Month, 1) : new DateTime(date.Year, date.Month, daysInMonth);
+                var boundaryDate = searchForward ? new DateTime(date.Year, date.Month, daysInMonth) : new DateTime(date.Year, date.Month, 1);
+
+                // Determine the scheduled day for this month
+                if (date == baseDate)
                 {
-                    // fire on Monday
-                    if (date.DayOfWeek == DayOfWeek.Monday)
+                    var scheduledDay = GetScheduledDay(securitySchedule, baseDate, offset, searchForward, boundaryDate);
+
+                    // Ensure the date is within our schedules range
+                    if (scheduledDay >= start && scheduledDay <= end)
                     {
-                        yield return date;
-                    }
-                }
-                else
-                {
-                    // skip Mondays and following days when market is closed
-                    if (date.DayOfWeek == DayOfWeek.Monday || skippedMarketClosedDay)
-                    {
-                        if (security.Exchange.Hours.IsDateOpen(date))
-                        {
-                            skippedMarketClosedDay = false;
-                            yield return date;
-                        }
-                        else
-                        {
-                            skippedMarketClosedDay = true;
-                        }
+                        yield return scheduledDay;
                     }
                 }
             }
         }
 
-        private static IEnumerable<DateTime> WeekEndIterator(Security security, DateTime start, DateTime end)
+        private static IEnumerable<DateTime> WeekIterator(SecurityExchangeHours securitySchedule, DateTime start, DateTime end, int offset, bool searchForward)
         {
-            foreach (var date in Time.EachDay(start, end))
+            // Determine the weekly base day and boundary to schedule off of
+            DayOfWeek weeklyBaseDay;
+            DayOfWeek weeklyBoundaryDay;
+            if (securitySchedule == null)
             {
-                if (date.DayOfWeek == DayOfWeek.Friday)
+                // No schedule means no security, set to open everyday
+                securitySchedule = SecurityExchangeHours.AlwaysOpen(TimeZones.NewYork);
+
+                // Searching forward Monday is baseDay, with boundary being the following Sunday
+                // Searching backward Friday is baseDay, with boundary being the previous Saturday
+                weeklyBaseDay = searchForward ? DayOfWeek.Monday : DayOfWeek.Friday;
+                weeklyBoundaryDay = searchForward ? DayOfWeek.Saturday + 1 : DayOfWeek.Sunday - 1;
+            }
+            else
+            {
+                // Fetch the securities schedule 
+                var weeklySchedule = securitySchedule.MarketHours.Values
+                    .Where(x => x.IsClosedAllDay == false).OrderBy(x => x.DayOfWeek).ToList();
+
+                // Determine our weekly base day and boundary for this security
+                weeklyBaseDay = searchForward ? weeklySchedule.First().DayOfWeek : weeklySchedule.Last().DayOfWeek;
+                weeklyBoundaryDay = searchForward ? weeklySchedule.Last().DayOfWeek : weeklySchedule.First().DayOfWeek;
+            }
+
+            // Determine the schedule for each week in this range
+            foreach (var date in Time.EachDay(start, end).Where(x => x.DayOfWeek == weeklyBaseDay))
+            {
+                var boundary = date.AddDays(weeklyBoundaryDay - weeklyBaseDay);
+                var scheduledDay = GetScheduledDay(securitySchedule, date, offset, searchForward, boundary);
+
+                // Ensure the date is within our schedules range
+                if (scheduledDay >= start && scheduledDay <= end)
                 {
-                    if (security == null)
-                    {
-                        // fire on Friday
-                        yield return date;
-                    }
-                    else
-                    {
-                        // find previous date when market is open
-                        var currentDate = date;
-                        while (!security.Exchange.Hours.IsDateOpen(currentDate))
-                        {
-                            currentDate = currentDate.AddDays(-1);
-                        }
-                        yield return currentDate;
-                    }
+                    yield return scheduledDay;
                 }
             }
         }

Common/Securities/ContractSecurityFilterUniverse.cs

@@ -0,0 +1,340 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+*/
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+using Python.Runtime;
+using QuantConnect.Data;
+
+namespace QuantConnect.Securities
+{
+    /// <summary>
+    /// Base class for contract symbols filtering universes.
+    /// Used by OptionFilterUniverse and FutureFilterUniverse
+    /// </summary>
+    public abstract class ContractSecurityFilterUniverse<T> : IDerivativeSecurityFilterUniverse
+    where T: ContractSecurityFilterUniverse<T>
+    {
+        /// <summary>
+        /// Defines listed contract types with Flags attribute
+        /// </summary>
+        [Flags]
+        protected enum ContractExpirationType : int
+        {
+            /// <summary>
+            /// Standard contracts
+            /// </summary>
+            Standard = 1,
+
+            /// <summary>
+            /// Non standard weekly contracts
+            /// </summary>
+            Weekly = 2
+        }
+
+        /// <summary>
+        /// Expiration Types allowed through the filter
+        /// Standards only by default
+        /// </summary>
+        protected ContractExpirationType Type = ContractExpirationType.Standard;
+
+        /// <summary>
+        /// The underlying price data
+        /// </summary>
+        protected BaseData UnderlyingInternal;
+
+        /// <summary>
+        /// All Symbols in this filter
+        /// Marked internal for use by extensions
+        /// </summary>
+        internal IEnumerable<Symbol> AllSymbols;
+
+        /// <summary>
+        /// Mark this filter dynamic for regular reapplying
+        /// Marked internal for use by extensions
+        /// </summary>
+        internal bool IsDynamicInternal;
+
+        /// <summary>
+        /// True if the universe is dynamic and filter needs to be reapplied
+        /// </summary>
+        public bool IsDynamic => IsDynamicInternal;
+
+        /// <summary>
+        /// The underlying price data
+        /// </summary>
+        public BaseData Underlying
+        {
+            get
+            {
+                // underlying value changes over time, so accessing it makes universe dynamic
+                IsDynamicInternal = true;
+                return UnderlyingInternal;
+            }
+        }
+
+        /// <summary>
+        /// Constructs ContractSecurityFilterUniverse
+        /// </summary>
+        protected ContractSecurityFilterUniverse()
+        {
+        }
+
+        /// <summary>
+        /// Constructs ContractSecurityFilterUniverse
+        /// </summary>
+        protected ContractSecurityFilterUniverse(IEnumerable<Symbol> allSymbols, BaseData underlying)
+        {
+            AllSymbols = allSymbols;
+            UnderlyingInternal = underlying;
+            Type = ContractExpirationType.Standard;
+            IsDynamicInternal = false;
+        }
+
+        /// <summary>
+        /// Function to determine if the given symbol is a standard contract
+        /// </summary>
+        /// <returns>True if standard type</returns>
+        protected abstract bool IsStandard(Symbol symbol);
+
+        /// <summary>
+        /// Returns universe, filtered by contract type
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        internal T ApplyTypesFilter()
+        {
+            // memoization map for ApplyTypesFilter()
+            var memoizedMap = new Dictionary<DateTime, bool>();
+
+            Func<Symbol, bool> memoizedIsStandardType = symbol =>
+            {
+                var dt = symbol.ID.Date;
+
+                bool result;
+                if (memoizedMap.TryGetValue(dt, out result))
+                    return result;
+                var res = IsStandard(symbol);
+                memoizedMap[dt] = res;
+
+                return res;
+            };
+
+            AllSymbols = AllSymbols.Where(x =>
+            {
+                switch (Type)
+                {
+                    case ContractExpirationType.Weekly:
+                        return !memoizedIsStandardType(x);
+                    case ContractExpirationType.Standard:
+                        return memoizedIsStandardType(x);
+                    case ContractExpirationType.Standard | ContractExpirationType.Weekly:
+                        return true;
+                    default:
+                        return false;
+                }
+            }).ToList();
+
+            return (T) this;
+        }
+
+        /// <summary>
+        /// Refreshes this filter universe
+        /// </summary>
+        /// <param name="allSymbols">All the contract symbols for the Universe</param>
+        /// <param name="underlying">The current underlying last data point</param>
+        public virtual void Refresh(IEnumerable<Symbol> allSymbols, BaseData underlying)
+        {
+            AllSymbols = allSymbols;
+            UnderlyingInternal = underlying;
+            Type = ContractExpirationType.Standard;
+            IsDynamicInternal = false;
+        }
+
+        /// <summary>
+        /// Sets universe of standard contracts (if any) as selection
+        /// Contracts by default are standards; only needed to switch back if changed
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        public T StandardsOnly()
+        {
+            Type = ContractExpirationType.Standard;
+            return (T)this;
+        }
+
+        /// <summary>
+        /// Includes universe of non-standard weeklys contracts (if any) into selection
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        public T IncludeWeeklys()
+        {
+            Type |= ContractExpirationType.Weekly;
+            return (T)this;
+        }
+
+        /// <summary>
+        /// Sets universe of weeklys contracts (if any) as selection
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        public T WeeklysOnly()
+        {
+            Type = ContractExpirationType.Weekly;
+            return (T)this;
+        }
+
+        /// <summary>
+        /// Returns front month contract
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        public virtual T FrontMonth()
+        {
+            var ordered = this.OrderBy(x => x.ID.Date).ToList();
+            if (ordered.Count == 0) return (T) this;
+            var frontMonth = ordered.TakeWhile(x => ordered[0].ID.Date == x.ID.Date);
+
+            AllSymbols = frontMonth.ToList();
+            return (T) this;
+        }
+
+        /// <summary>
+        /// Returns a list of back month contracts
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        public virtual T BackMonths()
+        {
+            var ordered = this.OrderBy(x => x.ID.Date).ToList();
+            if (ordered.Count == 0) return (T) this;
+            var backMonths = ordered.SkipWhile(x => ordered[0].ID.Date == x.ID.Date);
+
+            AllSymbols = backMonths.ToList();
+            return (T) this;
+        }
+
+        /// <summary>
+        /// Returns first of back month contracts
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        public T BackMonth()
+        {
+            return BackMonths().FrontMonth();
+        }
+
+        /// <summary>
+        /// Applies filter selecting options contracts based on a range of expiration dates relative to the current day
+        /// </summary>
+        /// <param name="minExpiry">The minimum time until expiry to include, for example, TimeSpan.FromDays(10)
+        /// would exclude contracts expiring in more than 10 days</param>
+        /// <param name="maxExpiry">The maximum time until expiry to include, for example, TimeSpan.FromDays(10)
+        /// would exclude contracts expiring in less than 10 days</param>
+        /// <returns>Universe with filter applied</returns>
+        public virtual T Expiration(TimeSpan minExpiry, TimeSpan maxExpiry)
+        {
+            if (UnderlyingInternal == null)
+            {
+                return (T) this;
+            }
+
+            if (maxExpiry > Time.MaxTimeSpan) maxExpiry = Time.MaxTimeSpan;
+
+            var minExpiryToDate = UnderlyingInternal.Time.Date + minExpiry;
+            var maxExpiryToDate = UnderlyingInternal.Time.Date + maxExpiry;
+
+            AllSymbols = AllSymbols
+                .Where(symbol => symbol.ID.Date >= minExpiryToDate && symbol.ID.Date <= maxExpiryToDate)
+                .ToList();
+
+            return (T) this;
+        }
+
+        /// <summary>
+        /// Applies filter selecting contracts based on a range of expiration dates relative to the current day
+        /// </summary>
+        /// <param name="minExpiryDays">The minimum time, expressed in days, until expiry to include, for example, 10
+        /// would exclude contracts expiring in more than 10 days</param>
+        /// <param name="maxExpiryDays">The maximum time, expressed in days, until expiry to include, for example, 10
+        /// would exclude contracts expiring in less than 10 days</param>
+        /// <returns>Universe with filter applied</returns>
+        public T Expiration(int minExpiryDays, int maxExpiryDays)
+        {
+            return Expiration(TimeSpan.FromDays(minExpiryDays), TimeSpan.FromDays(maxExpiryDays));
+        }
+
+        /// <summary>
+        /// Explicitly sets the selected contract symbols for this universe.
+        /// This overrides and and all other methods of selecting symbols assuming it is called last.
+        /// </summary>
+        /// <param name="contracts">The option contract symbol objects to select</param>
+        /// <returns>Universe with filter applied</returns>
+        public T Contracts(PyObject contracts)
+        {
+            AllSymbols = contracts.ConvertToSymbolEnumerable();
+            return (T) this;
+        }
+
+        /// <summary>
+        /// Explicitly sets the selected contract symbols for this universe.
+        /// This overrides and and all other methods of selecting symbols assuming it is called last.
+        /// </summary>
+        /// <param name="contracts">The option contract symbol objects to select</param>
+        /// <returns>Universe with filter applied</returns>
+        public T Contracts(IEnumerable<Symbol> contracts)
+        {
+            AllSymbols = contracts.ToList();
+            return (T) this;
+        }
+
+        /// <summary>
+        /// Sets a function used to filter the set of available contract filters. The input to the 'contractSelector'
+        /// function will be the already filtered list if any other filters have already been applied.
+        /// </summary>
+        /// <param name="contractSelector">The option contract symbol objects to select</param>
+        /// <returns>Universe with filter applied</returns>
+        public T Contracts(Func<IEnumerable<Symbol>, IEnumerable<Symbol>> contractSelector)
+        {
+            // force materialization using ToList
+            AllSymbols = contractSelector(AllSymbols).ToList();
+            return (T) this;
+        }
+
+        /// <summary>
+        /// Instructs the engine to only filter contracts on the first time step of each market day.
+        /// </summary>
+        /// <returns>Universe with filter applied</returns>
+        public T OnlyApplyFilterAtMarketOpen()
+        {
+            IsDynamicInternal = false;
+            return (T) this;
+        }
+
+        /// <summary>
+        /// IEnumerable interface method implementation
+        /// </summary>
+        /// <returns>IEnumerator of Symbols in Universe</returns>
+        public IEnumerator<Symbol> GetEnumerator()
+        {
+            return AllSymbols.GetEnumerator();
+        }
+
+        /// <summary>
+        /// IEnumerable interface method implementation
+        /// </summary>
+        IEnumerator IEnumerable.GetEnumerator()
+        {
+            return AllSymbols.GetEnumerator();
+        }
+    }
+}

Common/Securities/Equity/Equity.cs

@@ -53,7 +53,7 @@ namespace QuantConnect.Securities.Equity
                 new EquityExchange(exchangeHours),
                 securityCache,
                 new SecurityPortfolioModel(),
-                new ImmediateFillModel(),
+                new EquityFillModel(),
                 new InteractiveBrokersFeeModel(),
                 new ConstantSlippageModel(0m),
                 new ImmediateSettlementModel(),
@@ -84,7 +84,7 @@ namespace QuantConnect.Securities.Equity
                 new EquityExchange(exchangeHours),
                 new EquityCache(),
                 new SecurityPortfolioModel(),
-                new ImmediateFillModel(),
+                new EquityFillModel(),
                 new InteractiveBrokersFeeModel(),
                 new ConstantSlippageModel(0m),
                 new ImmediateSettlementModel(),

Common/Securities/Future/Future.cs

@@ -204,7 +204,8 @@ namespace QuantConnect.Securities.Future
             Func<IDerivativeSecurityFilterUniverse, IDerivativeSecurityFilterUniverse> func = universe =>
             {
                 var futureUniverse = universe as FutureFilterUniverse;
-                return universeFunc(futureUniverse);
+                var result = universeFunc(futureUniverse);
+                return result.ApplyTypesFilter();
             };
 
             ContractFilter = new FuncSecurityDerivativeFilter(func);

Common/Securities/Future/FutureFilterUniverse.cs

@@ -18,7 +18,7 @@ using System;
 using System.Collections.Generic;
 using QuantConnect.Data;
 using System.Linq;
-using System.Collections;
+using QuantConnect.Securities.Future;
 using QuantConnect.Util;
 
 namespace QuantConnect.Securities
@@ -26,182 +26,35 @@ namespace QuantConnect.Securities
     /// <summary>
     /// Represents futures symbols universe used in filtering.
     /// </summary>
-    public class FutureFilterUniverse : IDerivativeSecurityFilterUniverse
+    public class FutureFilterUniverse : ContractSecurityFilterUniverse<FutureFilterUniverse>
     {
-        internal IEnumerable<Symbol> _allSymbols;
-
-        /// <summary>
-        /// The underlying price data
-        /// </summary>
-        public BaseData Underlying
-        {
-            get { return _underlying; }
-        }
-
-        internal BaseData _underlying;
-
-        /// <summary>
-        /// True if the universe is dynamic and filter needs to be reapplied
-        /// </summary>
-        public bool IsDynamic
-        {
-            get
-            {
-                return _isDynamic;
-            }
-        }
-
-        internal bool _isDynamic;
-
         /// <summary>
         /// Constructs FutureFilterUniverse
         /// </summary>
         public FutureFilterUniverse(IEnumerable<Symbol> allSymbols, BaseData underlying)
+            : base(allSymbols, underlying)
         {
-            _allSymbols = allSymbols;
-            _underlying = underlying;
-            _isDynamic = false;
         }
 
         /// <summary>
-        /// Returns front month contract
+        /// Determine if the given Future contract symbol is standard
         /// </summary>
-        /// <returns></returns>
-        public FutureFilterUniverse FrontMonth()
+        /// <returns>True if contract is standard</returns>
+        protected override bool IsStandard(Symbol symbol)
         {
-            var ordered = this.OrderBy(x => x.ID.Date).ToList();
-            if (ordered.Count == 0) return this;
-            var frontMonth = ordered.TakeWhile(x => ordered[0].ID.Date == x.ID.Date);
-
-            _allSymbols = frontMonth.ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Returns a list of back month contracts
-        /// </summary>
-        /// <returns></returns>
-        public FutureFilterUniverse BackMonths()
-        {
-            var ordered = this.OrderBy(x => x.ID.Date).ToList();
-            if (ordered.Count == 0) return this;
-            var backMonths = ordered.SkipWhile(x => ordered[0].ID.Date == x.ID.Date);
-
-            _allSymbols = backMonths.ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Returns first of back month contracts
-        /// </summary>
-        /// <returns></returns>
-        public FutureFilterUniverse BackMonth()
-        {
-            return BackMonths().FrontMonth();
-        }
-
-        /// <summary>
-        /// Applies filter selecting futures contracts based on a range of expiration dates relative to the current day
-        /// </summary>
-        /// <param name="minExpiry">The minimum time until expiry to include, for example, TimeSpan.FromDays(10)
-        /// would exclude contracts expiring in more than 10 days</param>
-        /// <param name="maxExpiry">The maxmium time until expiry to include, for example, TimeSpan.FromDays(10)
-        /// would exclude contracts expiring in less than 10 days</param>
-        /// <returns></returns>
-        public FutureFilterUniverse Expiration(TimeSpan minExpiry, TimeSpan maxExpiry)
-        {
-            if (_underlying == null)
-            {
-                return this;
-            }
-
-            if (maxExpiry > Time.MaxTimeSpan) maxExpiry = Time.MaxTimeSpan;
-
-            var minExpiryToDate = _underlying.Time.Date + minExpiry;
-            var maxExpiryToDate = _underlying.Time.Date + maxExpiry;
-
-            var filtered =
-                   from symbol in _allSymbols
-                   let contract = symbol.ID
-                   where contract.Date >= minExpiryToDate
-                      && contract.Date <= maxExpiryToDate
-                   select symbol;
-
-            _allSymbols = filtered.ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Applies filter selecting futures contracts based on a range of expiration dates relative to the current day
-        /// </summary>
-        /// <param name="minExpiryDays">The minimum time, expressed in days, until expiry to include, for example, 10
-        /// would exclude contracts expiring in more than 10 days</param>
-        /// <param name="maxExpiryDays">The maximum time, expressed in days, until expiry to include, for example, 10
-        /// would exclude contracts expiring in less than 10 days</param>
-        /// <returns></returns>
-        public FutureFilterUniverse Expiration(int minExpiryDays, int maxExpiryDays)
-        {
-            return Expiration(TimeSpan.FromDays(minExpiryDays), TimeSpan.FromDays(maxExpiryDays));
+            return FutureSymbol.IsStandard(symbol);
         }
 
         /// <summary>
         /// Applies filter selecting futures contracts based on expiration cycles. See <see cref="FutureExpirationCycles"/> for details
         /// </summary>
-        /// <param name="months"></param>
-        /// <returns></returns>
+        /// <param name="months">Months to select contracts from</param>
+        /// <returns>Universe with filter applied</returns>
         public FutureFilterUniverse ExpirationCycle(int[] months)
         {
             var monthHashSet = months.ToHashSet();
             return this.Where(x => monthHashSet.Contains(x.ID.Date.Month));
         }
-
-        /// <summary>
-        /// Explicitly sets the selected contract symbols for this universe.
-        /// This overrides and and all other methods of selecting symbols assuming it is called last.
-        /// </summary>
-        /// <param name="contracts">The future contract symbol objects to select</param>
-        public FutureFilterUniverse Contracts(IEnumerable<Symbol> contracts)
-        {
-            _allSymbols = contracts.ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Sets a function used to filter the set of available contract filters. The input to the 'contractSelector'
-        /// function will be the already filtered list if any other filters have already been applied.
-        /// </summary>
-        /// <param name="contractSelector">The future contract symbol objects to select</param>
-        public FutureFilterUniverse Contracts(Func<IEnumerable<Symbol>, IEnumerable<Symbol>> contractSelector)
-        {
-            // force materialization using ToList
-            _allSymbols = contractSelector(_allSymbols).ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Instructs the engine to only filter options contracts on the first time step of each market day.
-        /// </summary>
-        public FutureFilterUniverse OnlyApplyFilterAtMarketOpen()
-        {
-            _isDynamic = false;
-            return this;
-        }
-
-        /// <summary>
-        /// IEnumerable interface method implementation
-        /// </summary>
-        public IEnumerator<Symbol> GetEnumerator()
-        {
-            return _allSymbols.GetEnumerator();
-        }
-
-        /// <summary>
-        /// IEnumerable interface method implementation
-        /// </summary>
-        IEnumerator IEnumerable.GetEnumerator()
-        {
-            return _allSymbols.GetEnumerator();
-        }
     }
 
     /// <summary>
@@ -212,33 +65,39 @@ namespace QuantConnect.Securities
         /// <summary>
         /// Filters universe
         /// </summary>
-        /// <param name="universe"></param>
-        /// <param name="predicate"></param>
-        /// <returns></returns>
+        /// <param name="universe">Universe to apply the filter too</param>
+        /// <param name="predicate">Bool function to determine which Symbol are filtered</param>
+        /// <returns><see cref="FutureFilterUniverse"/> with filter applied</returns>
         public static FutureFilterUniverse Where(this FutureFilterUniverse universe, Func<Symbol, bool> predicate)
         {
-            universe._allSymbols = universe._allSymbols.Where(predicate).ToList();
-            universe._isDynamic = true;
+            universe.AllSymbols = universe.AllSymbols.Where(predicate).ToList();
+            universe.IsDynamicInternal = true;
             return universe;
         }
 
         /// <summary>
         /// Maps universe
         /// </summary>
+        /// <param name="universe">Universe to apply the filter too</param>
+        /// <param name="mapFunc">Symbol function to determine which Symbols are filtered</param>
+        /// <returns><see cref="FutureFilterUniverse"/> with filter applied</returns>
         public static FutureFilterUniverse Select(this FutureFilterUniverse universe, Func<Symbol, Symbol> mapFunc)
         {
-            universe._allSymbols = universe._allSymbols.Select(mapFunc).ToList();
-            universe._isDynamic = true;
+            universe.AllSymbols = universe.AllSymbols.Select(mapFunc).ToList();
+            universe.IsDynamicInternal = true;
             return universe;
         }
 
         /// <summary>
         /// Binds universe
         /// </summary>
+        /// <param name="universe">Universe to apply the filter too</param>
+        /// <param name="mapFunc">Symbols function to determine which Symbols are filtered</param>
+        /// <returns><see cref="FutureFilterUniverse"/> with filter applied</returns>
         public static FutureFilterUniverse SelectMany(this FutureFilterUniverse universe, Func<Symbol, IEnumerable<Symbol>> mapFunc)
         {
-            universe._allSymbols = universe._allSymbols.SelectMany(mapFunc).ToList();
-            universe._isDynamic = true;
+            universe.AllSymbols = universe.AllSymbols.SelectMany(mapFunc).ToList();
+            universe.IsDynamicInternal = true;
             return universe;
         }
     }

Common/Securities/Future/FutureSymbol.cs

@@ -0,0 +1,61 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
+ * 
+ * Licensed under the Apache License, Version 2.0 (the "License"); 
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+*/
+
+using QuantConnect.Logging;
+
+namespace QuantConnect.Securities.Future
+{
+    /// <summary>
+    /// Static class contains common utility methods specific to symbols representing the future contracts
+    /// </summary>
+    public static class FutureSymbol
+    {
+        /// <summary>
+        /// Determine if a given Futures contract is a standard contract.
+        /// </summary>
+        /// <param name="symbol">Future symbol</param>
+        /// <returns>True if symbol expiration matches standard expiration</returns>
+        public static bool IsStandard(Symbol symbol)
+        {
+            var contractExpirationDate = symbol.ID.Date;
+
+            try
+            {
+                // Use our FutureExpiryFunctions to determine standard contracts dates.
+                var expiryFunction = FuturesExpiryFunctions.FuturesExpiryFunction(symbol);
+                var standardExpirationDate = expiryFunction(contractExpirationDate);
+
+                // Return true if the dates match
+                return contractExpirationDate.Date == standardExpirationDate.Date;
+            }
+            catch
+            {
+                Log.Error($"Could not find standard date for {symbol}, will be classified as standard");
+                return true;
+            }
+        }
+
+        /// <summary>
+        /// Returns true if the future contract is a weekly contract
+        /// </summary>
+        /// <param name="symbol">Future symbol</param>
+        /// <returns>True if symbol is non-standard contract</returns>
+        public static bool IsWeekly(Symbol symbol)
+        {
+            return !IsStandard(symbol);
+        }
+    }
+}

Common/Securities/Option/Option.cs

@@ -458,7 +458,7 @@ namespace QuantConnect.Securities.Option
             {
                 var optionUniverse = universe as OptionFilterUniverse;
                 var result = universeFunc(optionUniverse);
-                return result.ApplyOptionTypesFilter();
+                return result.ApplyTypesFilter();
             });
         }
 
@@ -496,7 +496,7 @@ namespace QuantConnect.Securities.Option
                             $"filter function is not a valid argument, please return either a OptionFilterUniverse or a list of symbols");
                     }
                 }
-                return optionUniverse.ApplyOptionTypesFilter();
+                return optionUniverse.ApplyTypesFilter();
             });
         }
 

Common/Securities/Option/OptionFilterUniverse.cs

@@ -18,8 +18,6 @@ using System;
 using System.Collections.Generic;
 using QuantConnect.Data;
 using System.Linq;
-using System.Collections;
-using Python.Runtime;
 using QuantConnect.Securities.Option;
 
 namespace QuantConnect.Securities
@@ -27,50 +25,8 @@ namespace QuantConnect.Securities
     /// <summary>
     /// Represents options symbols universe used in filtering.
     /// </summary>
-    public class OptionFilterUniverse : IDerivativeSecurityFilterUniverse
+    public class OptionFilterUniverse : ContractSecurityFilterUniverse<OptionFilterUniverse>
     {
-        /// <summary>
-        /// Defines listed option types
-        /// </summary>
-        public enum Type : int
-        {
-            /// <summary>
-            /// Listed stock options that expire 3rd Friday of the month
-            /// </summary>
-            Standard = 1,
-
-            /// <summary>
-            /// Weeklys options that expire every week
-            /// These are options listed with approximately one week to expiration
-            /// </summary>
-            Weeklys = 2
-        }
-
-        internal IEnumerable<Symbol> _allSymbols;
-
-        /// <summary>
-        /// The underlying price data
-        /// </summary>
-        public BaseData Underlying
-        {
-            get
-            {
-                // underlying value changes over time, so accessing it makes universe dynamic
-                _isDynamic = true;
-                return _underlying;
-            }
-        }
-
-        private BaseData _underlying;
-
-        /// <summary>
-        /// True if the universe is dynamic and filter needs to be reapplied
-        /// </summary>
-        public bool IsDynamic => _isDynamic;
-
-        internal bool _isDynamic;
-
-        private Type _type = Type.Standard;
         // Fields used in relative strikes filter
         private List<decimal> _uniqueStrikes;
         private bool _refreshUniqueStrikes;
@@ -85,9 +41,10 @@ namespace QuantConnect.Securities
         /// <summary>
         /// Constructs OptionFilterUniverse
         /// </summary>
-        public OptionFilterUniverse(IEnumerable<Symbol> allSymbols, BaseData underlying)
+        public OptionFilterUniverse(IEnumerable<Symbol> allSymbols, BaseData underlying) 
+            : base(allSymbols, underlying)
         {
-            Refresh(allSymbols, underlying, exchangeDateChange: true);
+            _refreshUniqueStrikes = true;
         }
 
         /// <summary>
@@ -98,108 +55,17 @@ namespace QuantConnect.Securities
         /// <param name="exchangeDateChange">True if the exchange data has chanced since the last call or construction</param>
         public void Refresh(IEnumerable<Symbol> allSymbols, BaseData underlying, bool exchangeDateChange = true)
         {
-            _allSymbols = allSymbols;
-            _underlying = underlying;
-            _type = Type.Standard;
-            _isDynamic = false;
+            base.Refresh(allSymbols, underlying);
             _refreshUniqueStrikes = exchangeDateChange;
         }
 
         /// <summary>
-        /// Includes universe of weeklys options (if any) into selection
+        /// Determine if the given Option contract symbol is standard
         /// </summary>
-        /// <returns></returns>
-        public OptionFilterUniverse IncludeWeeklys()
+        /// <returns>True if standard</returns>
+        protected override bool IsStandard(Symbol symbol)
         {
-            _type |= Type.Weeklys;
-            return this;
-        }
-
-        /// <summary>
-        /// Sets universe of weeklys options (if any) as a selection
-        /// </summary>
-        /// <returns></returns>
-        public OptionFilterUniverse WeeklysOnly()
-        {
-            _type = Type.Weeklys;
-            return this;
-        }
-
-        /// <summary>
-        /// Returns universe, filtered by option type
-        /// </summary>
-        /// <returns></returns>
-        internal OptionFilterUniverse ApplyOptionTypesFilter()
-        {
-            // memoization map for ApplyOptionTypesFilter()
-            var memoizedMap = new Dictionary<DateTime, bool>();
-
-            Func<Symbol, bool> memoizedIsStandardType = symbol =>
-            {
-                var dt = symbol.ID.Date;
-
-                bool result;
-                if (memoizedMap.TryGetValue(dt, out result))
-                    return result;
-                var res = OptionSymbol.IsStandard(symbol);
-                memoizedMap[dt] = res;
-
-                return res;
-            };
-
-            _allSymbols = _allSymbols.Where(x =>
-            {
-                switch (_type)
-                {
-                    case Type.Weeklys:
-                        return !memoizedIsStandardType(x);
-                    case Type.Standard:
-                        return memoizedIsStandardType(x);
-                    case Type.Standard | Type.Weeklys:
-                        return true;
-                    default:
-                        return false;
-                }
-            }).ToList();
-
-            return this;
-        }
-
-        /// <summary>
-        /// Returns front month contract
-        /// </summary>
-        /// <returns></returns>
-        public OptionFilterUniverse FrontMonth()
-        {
-            var ordered = this.OrderBy(x => x.ID.Date).ToList();
-            if (ordered.Count == 0) return this;
-            var frontMonth = ordered.TakeWhile(x => ordered[0].ID.Date == x.ID.Date);
-
-            _allSymbols = frontMonth.ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Returns a list of back month contracts
-        /// </summary>
-        /// <returns></returns>
-        public OptionFilterUniverse BackMonths()
-        {
-            var ordered = this.OrderBy(x => x.ID.Date).ToList();
-            if (ordered.Count == 0) return this;
-            var backMonths = ordered.SkipWhile(x => ordered[0].ID.Date == x.ID.Date);
-
-            _allSymbols = backMonths.ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Returns first of back month contracts
-        /// </summary>
-        /// <returns></returns>
-        public OptionFilterUniverse BackMonth()
-        {
-            return BackMonths().FrontMonth();
+            return OptionSymbol.IsStandard(symbol);
         }
 
         /// <summary>
@@ -207,10 +73,10 @@ namespace QuantConnect.Securities
         /// </summary>
         /// <param name="minStrike">The minimum strike relative to the underlying price, for example, -1 would filter out contracts further than 1 strike below market price</param>
         /// <param name="maxStrike">The maximum strike relative to the underlying price, for example, +1 would filter out contracts further than 1 strike above market price</param>
-        /// <returns></returns>
+        /// <returns>Universe with filter applied</returns>
         public OptionFilterUniverse Strikes(int minStrike, int maxStrike)
         {
-            if (_underlying == null)
+            if (UnderlyingInternal == null)
             {
                 return this;
             }
@@ -218,7 +84,7 @@ namespace QuantConnect.Securities
             if (_refreshUniqueStrikes || _uniqueStrikes == null)
             {
                 // each day we need to recompute the unique strikes list
-                _uniqueStrikes = _allSymbols.Select(x => x.ID.StrikePrice)
+                _uniqueStrikes = AllSymbols.Select(x => x.ID.StrikePrice)
                     .Distinct()
                     .OrderBy(strikePrice => strikePrice)
                     .ToList();
@@ -226,11 +92,11 @@ namespace QuantConnect.Securities
             }
 
             // new universe is dynamic
-            _isDynamic = true;
+            IsDynamicInternal = true;
 
             // find the current price in the list of strikes
             var exactPriceFound = true;
-            var index = _uniqueStrikes.BinarySearch(_underlying.Price);
+            var index = _uniqueStrikes.BinarySearch(UnderlyingInternal.Price);
 
             // Return value of BinarySearch (from MSDN):
             // The zero-based index of item in the sorted List<T>, if item is found;
@@ -244,7 +110,7 @@ namespace QuantConnect.Securities
                 if (index == ~_uniqueStrikes.Count)
                 {
                     // there is no greater price, return empty
-                    _allSymbols = Enumerable.Empty<Symbol>();
+                    AllSymbols = Enumerable.Empty<Symbol>();
                     return this;
                 }
 
@@ -274,14 +140,14 @@ namespace QuantConnect.Securities
             else if (indexMinPrice >= _uniqueStrikes.Count)
             {
                 // price out of range: return empty
-                _allSymbols = Enumerable.Empty<Symbol>();
+                AllSymbols = Enumerable.Empty<Symbol>();
                 return this;
             }
 
             if (indexMaxPrice < 0)
             {
                 // price out of range: return empty
-                _allSymbols = Enumerable.Empty<Symbol>();
+                AllSymbols = Enumerable.Empty<Symbol>();
                 return this;
             }
             if (indexMaxPrice >= _uniqueStrikes.Count)
@@ -292,7 +158,7 @@ namespace QuantConnect.Securities
             var minPrice = _uniqueStrikes[indexMinPrice];
             var maxPrice = _uniqueStrikes[indexMaxPrice];
 
-            _allSymbols = _allSymbols
+            AllSymbols = AllSymbols
                 .Where(symbol =>
                     {
                         var price = symbol.ID.StrikePrice;
@@ -303,83 +169,10 @@ namespace QuantConnect.Securities
             return this;
         }
 
-        /// <summary>
-        /// Applies filter selecting options contracts based on a range of expiration dates relative to the current day
-        /// </summary>
-        /// <param name="minExpiry">The minimum time until expiry to include, for example, TimeSpan.FromDays(10)
-        /// would exclude contracts expiring in more than 10 days</param>
-        /// <param name="maxExpiry">The maxmium time until expiry to include, for example, TimeSpan.FromDays(10)
-        /// would exclude contracts expiring in less than 10 days</param>
-        /// <returns></returns>
-        public OptionFilterUniverse Expiration(TimeSpan minExpiry, TimeSpan maxExpiry)
-        {
-            if (_underlying == null)
-            {
-                return this;
-            }
-
-            if (maxExpiry > Time.MaxTimeSpan) maxExpiry = Time.MaxTimeSpan;
-
-            var minExpiryToDate = _underlying.Time.Date + minExpiry;
-            var maxExpiryToDate = _underlying.Time.Date + maxExpiry;
-
-            _allSymbols = _allSymbols
-                .Where(symbol => symbol.ID.Date >= minExpiryToDate && symbol.ID.Date <= maxExpiryToDate)
-                .ToList();
-
-            return this;
-        }
-
-        /// <summary>
-        /// Applies filter selecting options contracts based on a range of expiration dates relative to the current day
-        /// </summary>
-        /// <param name="minExpiryDays">The minimum time, expressed in days, until expiry to include, for example, 10
-        /// would exclude contracts expiring in more than 10 days</param>
-        /// <param name="maxExpiryDays">The maximum time, expressed in days, until expiry to include, for example, 10
-        /// would exclude contracts expiring in less than 10 days</param>
-        /// <returns></returns>
-        public OptionFilterUniverse Expiration(int minExpiryDays, int maxExpiryDays)
-        {
-            return Expiration(TimeSpan.FromDays(minExpiryDays), TimeSpan.FromDays(maxExpiryDays));
-        }
-
-        /// <summary>
-        /// Explicitly sets the selected contract symbols for this universe.
-        /// This overrides and and all other methods of selecting symbols assuming it is called last.
-        /// </summary>
-        /// <param name="contracts">The option contract symbol objects to select</param>
-        public OptionFilterUniverse Contracts(PyObject contracts)
-        {
-            _allSymbols = contracts.ConvertToSymbolEnumerable();
-            return this;
-        }
-
-        /// <summary>
-        /// Explicitly sets the selected contract symbols for this universe.
-        /// This overrides and and all other methods of selecting symbols assuming it is called last.
-        /// </summary>
-        /// <param name="contracts">The option contract symbol objects to select</param>
-        public OptionFilterUniverse Contracts(IEnumerable<Symbol> contracts)
-        {
-            _allSymbols = contracts.ToList();
-            return this;
-        }
-
-        /// <summary>
-        /// Sets a function used to filter the set of available contract filters. The input to the 'contractSelector'
-        /// function will be the already filtered list if any other filters have already been applied.
-        /// </summary>
-        /// <param name="contractSelector">The option contract symbol objects to select</param>
-        public OptionFilterUniverse Contracts(Func<IEnumerable<Symbol>, IEnumerable<Symbol>> contractSelector)
-        {
-            // force materialization using ToList
-            _allSymbols = contractSelector(_allSymbols).ToList();
-            return this;
-        }
-
         /// <summary>
         /// Sets universe of call options (if any) as a selection
         /// </summary>
+        /// <returns>Universe with filter applied</returns>
         public OptionFilterUniverse CallsOnly()
         {
             return Contracts(contracts => contracts.Where(x => x.ID.OptionRight == OptionRight.Call));
@@ -388,35 +181,11 @@ namespace QuantConnect.Securities
         /// <summary>
         /// Sets universe of put options (if any) as a selection
         /// </summary>
+        /// <returns>Universe with filter applied</returns>
         public OptionFilterUniverse PutsOnly()
         {
             return Contracts(contracts => contracts.Where(x => x.ID.OptionRight == OptionRight.Put));
         }
-
-        /// <summary>
-        /// Instructs the engine to only filter options contracts on the first time step of each market day.
-        /// </summary>
-        public OptionFilterUniverse OnlyApplyFilterAtMarketOpen()
-        {
-            _isDynamic = false;
-            return this;
-        }
-
-        /// <summary>
-        /// IEnumerable interface method implementation
-        /// </summary>
-        public IEnumerator<Symbol> GetEnumerator()
-        {
-            return _allSymbols.GetEnumerator();
-        }
-
-        /// <summary>
-        /// IEnumerable interface method implementation
-        /// </summary>
-        IEnumerator IEnumerable.GetEnumerator()
-        {
-            return _allSymbols.GetEnumerator();
-        }
     }
 
     /// <summary>
@@ -427,43 +196,52 @@ namespace QuantConnect.Securities
         /// <summary>
         /// Filters universe
         /// </summary>
-        /// <param name="universe"></param>
-        /// <param name="predicate"></param>
-        /// <returns></returns>
+        /// <param name="universe">Universe to apply the filter too</param>
+        /// <param name="predicate">Bool function to determine which Symbol are filtered</param>
+        /// <returns>Universe with filter applied</returns>
         public static OptionFilterUniverse Where(this OptionFilterUniverse universe, Func<Symbol, bool> predicate)
         {
-            universe._allSymbols = universe._allSymbols.Where(predicate).ToList();
-            universe._isDynamic = true;
+            universe.AllSymbols = universe.AllSymbols.Where(predicate).ToList();
+            universe.IsDynamicInternal = true;
             return universe;
         }
 
         /// <summary>
         /// Maps universe
         /// </summary>
+        /// <param name="universe">Universe to apply the filter too</param>
+        /// <param name="mapFunc">Symbol function to determine which Symbols are filtered</param>
+        /// <returns>Universe with filter applied</returns>
         public static OptionFilterUniverse Select(this OptionFilterUniverse universe, Func<Symbol, Symbol> mapFunc)
         {
-            universe._allSymbols = universe._allSymbols.Select(mapFunc).ToList();
-            universe._isDynamic = true;
+            universe.AllSymbols = universe.AllSymbols.Select(mapFunc).ToList();
+            universe.IsDynamicInternal = true;
             return universe;
         }
 
         /// <summary>
         /// Binds universe
         /// </summary>
+        /// <param name="universe">Universe to apply the filter too</param>
+        /// <param name="mapFunc">Symbol function to determine which Symbols are filtered</param>
+        /// <returns>Universe with filter applied</returns>
         public static OptionFilterUniverse SelectMany(this OptionFilterUniverse universe, Func<Symbol, IEnumerable<Symbol>> mapFunc)
         {
-            universe._allSymbols = universe._allSymbols.SelectMany(mapFunc).ToList();
-            universe._isDynamic = true;
+            universe.AllSymbols = universe.AllSymbols.SelectMany(mapFunc).ToList();
+            universe.IsDynamicInternal = true;
             return universe;
         }
 
         /// <summary>
         /// Updates universe to only contain the symbols in the list
         /// </summary>
+        /// <param name="universe">Universe to apply the filter too</param>
+        /// <param name="filterList">List of Symbols to keep in the Universe</param>
+        /// <returns>Universe with filter applied</returns>
         public static OptionFilterUniverse WhereContains(this OptionFilterUniverse universe, List<Symbol> filterList)
         {
-            universe._allSymbols = universe._allSymbols.Where(filterList.Contains).ToList();
-            universe._isDynamic = true;
+            universe.AllSymbols = universe.AllSymbols.Where(filterList.Contains).ToList();
+            universe.IsDynamicInternal = true;
             return universe;
         }
     }

Common/Securities/SecurityPortfolioManager.cs

@@ -541,8 +541,8 @@ namespace QuantConnect.Securities
                 if (accountCurrency != CashBook.AccountCurrency)
                 {
                     Log.Trace("SecurityPortfolioManager.SetAccountCurrency():" +
-                              $" account currency has already been set to {CashBook.AccountCurrency}." +
-                              $" Will ignore new value {accountCurrency}");
+                        $" account currency has already been set to {CashBook.AccountCurrency}." +
+                        $" Will ignore new value {accountCurrency}");
                 }
                 return;
             }

Common/Util/Composer.cs

@@ -24,6 +24,7 @@ using System.ComponentModel.Composition.ReflectionModel;
 using System.IO;
 using System.Linq;
 using System.Reflection;
+using System.Threading;
 using System.Threading.Tasks;
 using QuantConnect.Configuration;
 using QuantConnect.Logging;
@@ -81,7 +82,11 @@ namespace QuantConnect.Util
                 }
                 catch (Exception exception)
                 {
-                    Log.Error(exception);
+                    // ThreadAbortException is triggered when we shutdown ignore the error log
+                    if (!(exception is ThreadAbortException))
+                    {
+                        Log.Error(exception);
+                    }
                 }
                 return new List<ComposablePartDefinition>();
             });

Common/packages.config

@@ -7,6 +7,7 @@
   <package id="Microsoft.CodeAnalysis.FxCopAnalyzers" version="2.9.3" targetFramework="net452" />
   <package id="Microsoft.CodeAnalysis.VersionCheckAnalyzer" version="2.9.3" targetFramework="net452" />
   <package id="Microsoft.CodeQuality.Analyzers" version="2.9.3" targetFramework="net452" />
+  <package id="Microsoft.IO.RecyclableMemoryStream" version="1.3.5" targetFramework="net462" />
   <package id="Microsoft.NetCore.Analyzers" version="2.9.3" targetFramework="net452" />
   <package id="Microsoft.NetFramework.Analyzers" version="2.9.3" targetFramework="net452" />
   <package id="Newtonsoft.Json" version="10.0.3" targetFramework="net452" />

Configuration/ToolboxArgumentParser.cs

@@ -16,6 +16,7 @@
 
 using System;
 using System.Collections.Generic;
+using System.Linq;
 using Microsoft.Extensions.CommandLineUtils;
 
 namespace QuantConnect.Configuration
@@ -87,5 +88,15 @@ namespace QuantConnect.Configuration
         {
             return ApplicationParser.Parse(ApplicationName, ApplicationDescription, ApplicationHelpText, args, Options);
         }
+
+        /// <summary>
+        /// Helper method to get the tickers from the provided options
+        /// </summary>
+        public static List<string> GetTickers(Dictionary<string, object> optionsObject)
+        {
+            return optionsObject.ContainsKey("tickers")
+                ? (optionsObject["tickers"] as Dictionary<string, string>)?.Keys.ToList()
+                : new List<string>();
+        }
     }
 }

Data/equity/usa/factor_files/wm.csv

@@ -1,66 +1,76 @@
-19980102,0.6351976,1
-19980925,0.6351976,1
-19981228,0.6353279,1
-19990927,0.6354702,1
-20000926,0.6358027,1
-20010924,0.6361571,1
-20020925,0.6364034,1
-20030925,0.6366759,1
-20040225,0.6369160,1
-20040526,0.6410572,1
-20040827,0.6452403,1
-20041126,0.6495540,1
-20050224,0.6536655,1
-20050526,0.6581396,1
-20050829,0.6626276,1
-20051128,0.6675017,1
-20060301,0.6719521,1
-20060531,0.6763674,1
-20060830,0.6804556,1
-20061129,0.6849006,1
-20070307,0.6889898,1
-20070530,0.6939215,1
-20070829,0.6982113,1
-20071128,0.7027001,1
-20080305,0.7075746,1
-20080528,0.7132636,1
-20080827,0.7184530,1
-20081125,0.7240543,1
-20090304,0.7306454,1
-20090527,0.7393580,1
-20090902,0.7470844,1
-20091127,0.7545041,1
-20100226,0.7611812,1
-20100526,0.7685128,1
-20100901,0.7761008,1
-20101126,0.7833586,1
-20110308,0.7904896,1
-20110527,0.7977226,1
-20110831,0.8047618,1
-20111125,0.8131297,1
-20120306,0.8223541,1
-20120605,0.8308495,1
-20120904,0.8401118,1
-20121123,0.8487804,1
-20130306,0.8581948,1
-20130604,0.8666930,1
-20130903,0.8743351,1
-20131129,0.8823586,1
-20140305,0.8894657,1
-20140603,0.8976080,1
-20140902,0.9052089,1
-20141202,0.9124506,1
-20150304,0.9196092,1
-20150602,0.9260812,1
-20150909,0.9333315,1
-20151201,0.9405493,1
-20160302,0.9473158,1
-20160531,0.9541706,1
-20160906,0.9606336,1
-20161129,0.9667643,1
-20170307,0.9724284,1
-20170606,0.9782142,1
-20170906,0.9839097,1
-20171129,0.9894333,1
-20180307,0.9945500,1
+19980102,0.6051633,1
+19980925,0.6051633,1,48.81
+19981228,0.6052874,1,44.44
+19990927,0.6054230,1,19.19
+20000926,0.6057398,1,17.94
+20010924,0.6060774,1,25.82
+20020925,0.6063120,1,23.43
+20030925,0.6065717,1,26.54
+20040225,0.6068004,1,29.14
+20040526,0.6107458,1,29
+20040827,0.6147311,1,28.32
+20041126,0.6188408,1,29.89
+20050224,0.6227580,1,29.41
+20050526,0.6270205,1,29.53
+20050829,0.6312962,1,27.41
+20051128,0.6359398,1,30.25
+20060301,0.6401798,1,33.7
+20060531,0.6443863,1,36.62
+20060830,0.6482812,1,33.88
+20061129,0.6525160,1,37.03
+20070307,0.6564118,1,33.76
+20070530,0.6611103,1,39.06
+20070829,0.6651973,1,37.55
+20071128,0.6694739,1,34.84
+20080305,0.6741179,1,33.83
+20080528,0.6795379,1,37.43
+20080827,0.6844819,1,34.9
+20081125,0.6898184,1,29.92
+20090304,0.6960979,1,24.62
+20090527,0.7043985,1,28.06
+20090902,0.7117595,1,29.49
+20091127,0.7188284,1,33.09
+20100226,0.7251898,1,33.02
+20100526,0.7321748,1,32.22
+20100901,0.7394039,1,34
+20101126,0.7463185,1,34.92
+20110308,0.7531123,1,37.5
+20110527,0.7600033,1,38.87
+20110831,0.7667097,1,33.04
+20111125,0.7746819,1,30.31
+20120306,0.7834701,1,34.72
+20120605,0.7915638,1,32.2
+20120904,0.8003881,1,34.76
+20121123,0.8086468,1,32.35
+20130306,0.8176160,1,37.23
+20130604,0.8257124,1,41.77
+20130903,0.8329932,1,40.15
+20131129,0.8406373,1,45.68
+20140305,0.8474085,1,41.34
+20140603,0.8551658,1,44.67
+20140902,0.8624073,1,47.26
+20141202,0.8693065,1,48.82
+20150304,0.8761267,1,55.07
+20150602,0.8822926,1,49.56
+20150909,0.8892002,1,50.16
+20151201,0.8960766,1,53.92
+20160302,0.9025231,1,57.07
+20160531,0.9090538,1,60.95
+20160906,0.9152112,1,64.65
+20161129,0.9210520,1,70.39
+20170307,0.9264482,1,72.7
+20170606,0.9319605,1,73.41
+20170906,0.9373867,1,76.13
+20171129,0.9426491,1,82.62
+20180307,0.9475239,1,85.3
+20180606,0.9527162,1,83.78
+20180905,0.9580335,1,91.54
+20181128,0.9629248,1,92.81
+20190306,0.9677736,1,99.81
+20190605,0.9727637,1,113.44
+20190904,0.9771744,1,120.68
+20191204,0.9813380,1,112.86
+20200304,0.9858137,1,120.29
+20200603,0.9903012,1,110.21
+20200902,0.9952227,1,114.15
 20501231,1.0000000,1

Data/symbol-properties/symbol-properties-database.csv

@@ -125,7 +125,7 @@ cme,CB,future,Cash-settled Butter Futures,USD,20000.0,0.00025,1.0
 cme,CJY,future,Canadian Dollar/Japanese Yen Futures,JPY,200000.0,0.01,1.0
 cme,CNH,future,Standard-Size USD/Offshore RMB (CNH) Futures,CNY,100000.0,0.0001,1.0
 cme,CSC,future,Cash-Settled Cheese Futures,USD,20000.0,0.001,1.0
-cme,DC,future,Class III Milk Futures,USD,200000.0,0.0001,1.0
+cme,DC,future,Class III Milk Futures,USD,2000.0,0.01,1.0
 cme,DY,future,Dry Whey Futures,USD,44000.0,0.00025,1.0
 cme,E7,future,E-mini Euro FX Futures,USD,62500.0,0.0001,1.0
 cme,EAD,future,Euro/Australian Dollar Futures,AUD,125000.0,0.0001,1.0
@@ -135,7 +135,7 @@ cme,EMD,future,E-mini S&P MidCap 400 Futures,USD,100.0,0.1,1.0
 cme,ES,future,E-mini S&P 500 Futures,USD,50.0,0.25,1.0
 cme,ESK,future,Euro/Swedish Krona Futures,SEK,125000.0,0.0005,1.0
 cme,GD,future,S&P-GSCI Commodity Index Futures,USD,250.0,0.05,1.0
-cme,GDK,future,Class IV Milk Futures,USD,200000.0,0.0001,1.0
+cme,GDK,future,Class IV Milk Futures,USD,2000.0,0.01,1.0
 cme,GE,future,Eurodollar Futures,USD,2500.0,0.0025,1.0
 cme,GF,future,Feeder Cattle Futures,USD,500.0,0.025,1.0
 cme,GNF,future,Nonfat Dry Milk Futures,USD,44000.0,0.00025,1.0

DockerfileJupyter

@@ -14,9 +14,11 @@ RUN wget --quiet https://github.com/krallin/tini/releases/download/v0.10.0/tini
     mv tini /usr/local/bin/tini && \
     chmod +x /usr/local/bin/tini
 
-# Install Lean/PythonToolbox
-RUN git clone https://github.com/QuantConnect/Lean.git && cd Lean/PythonToolbox && \
-    python setup.py install && cd ../.. && rm -irf Lean
+# Clone Lean; Copy Python startup file to profile; Install Lean/PythonToolbox; Remove extra files
+RUN git clone https://github.com/QuantConnect/Lean.git && \
+    mkdir -p /root/.ipython/profile_default/startup/ && cp -f Lean/Research/start.py /root/.ipython/profile_default/startup/ && \
+    cd Lean/PythonToolbox && python setup.py install \
+    && cd ../.. && rm -irf Lean
 
 RUN conda install -y -c conda-forge notebook=6.0.3
 

Engine/AlgorithmManager.cs

@@ -499,6 +499,7 @@ namespace QuantConnect.Lean.Engine
                         var timeKeeper = algorithm.TimeKeeper;
                         foreach (var update in timeSlice.ConsolidatorUpdateData)
                         {
+                            var localTime = timeKeeper.GetLocalTimeKeeper(update.Target.ExchangeTimeZone).LocalTime;
                             var consolidators = update.Target.Consolidators;
                             foreach (var consolidator in consolidators)
                             {
@@ -512,7 +513,7 @@ namespace QuantConnect.Lean.Engine
                                 }
 
                                 // scan for time after we've pumped all the data through for this consolidator
-                                consolidator.Scan(timeKeeper.GetLocalTimeKeeper(update.Target.ExchangeTimeZone).LocalTime);
+                                consolidator.Scan(localTime);
                             }
                         }
                     }
@@ -586,16 +587,6 @@ namespace QuantConnect.Lean.Engine
                 //After we've fired all other events in this second, fire the pricing events:
                 try
                 {
-
-                    // TODO: For backwards compatibility only. Remove in 2017
-                    // For compatibility with Forex Trade data, moving
-                    if (timeSlice.Slice.QuoteBars.Count > 0)
-                    {
-                        foreach (var tradeBar in timeSlice.Slice.QuoteBars.Where(x => x.Key.ID.SecurityType == SecurityType.Forex))
-                        {
-                            timeSlice.Slice.Bars.Add(tradeBar.Value.Collapse());
-                        }
-                    }
                     if (hasOnDataTradeBars && timeSlice.Slice.Bars.Count > 0) methodInvokers[typeof(TradeBars)](algorithm, timeSlice.Slice.Bars);
                     if (hasOnDataQuoteBars && timeSlice.Slice.QuoteBars.Count > 0) methodInvokers[typeof(QuoteBars)](algorithm, timeSlice.Slice.QuoteBars);
                     if (hasOnDataOptionChains && timeSlice.Slice.OptionChains.Count > 0) methodInvokers[typeof(OptionChains)](algorithm, timeSlice.Slice.OptionChains);

Engine/DataFeeds/Enumerators/AuxiliaryDataEnumerator.cs

@@ -136,43 +136,5 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators
             get;
             private set;
         }
-
-        /// <summary>
-        /// Un-normalizes the PreviousUnderlyingData.Value
-        /// </summary>
-        public static decimal GetRawClose(decimal price, SubscriptionDataConfig config)
-        {
-            return GetRawValue(price, config.SumOfDividends, config.PriceScaleFactor, config.DataNormalizationMode);
-        }
-
-        /// <summary>
-        /// Un-normalizes a price
-        /// </summary>
-        private static decimal GetRawValue(decimal price,
-            decimal sumOfDividends,
-            decimal priceScaleFactor,
-            DataNormalizationMode dataNormalizationMode)
-        {
-            switch (dataNormalizationMode)
-            {
-                case DataNormalizationMode.Raw:
-                    break;
-
-                case DataNormalizationMode.SplitAdjusted:
-                case DataNormalizationMode.Adjusted:
-                    // we need to 'unscale' the price
-                    price = price / priceScaleFactor;
-                    break;
-
-                case DataNormalizationMode.TotalReturn:
-                    // we need to remove the dividends since we've been accumulating them in the price
-                    price = (price - sumOfDividends) / priceScaleFactor;
-                    break;
-
-                default:
-                    throw new ArgumentOutOfRangeException();
-            }
-            return price;
-        }
     }
 }

Engine/DataFeeds/Enumerators/DividendEventProvider.cs

@@ -30,6 +30,7 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators
         // we set the price factor ratio when we encounter a dividend in the factor file
         // and on the next trading day we use this data to produce the dividend instance
         private decimal? _priceFactorRatio;
+        private decimal _referencePrice;
         private FactorFile _factorFile;
         private MapFile _mapFile;
         private SubscriptionDataConfig _config;
@@ -64,27 +65,32 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators
             {
                 if (_priceFactorRatio != null)
                 {
-                    var close = AuxiliaryDataEnumerator.GetRawClose(
-                        eventArgs.LastBaseData?.Price ?? 0,
-                        _config);
+                    if (_referencePrice == 0)
+                    {
+                        throw new InvalidOperationException($"Zero reference price for {_config.Symbol} dividend at {eventArgs.Date}");
+                    }
+
                     var baseData = Dividend.Create(
                         _config.Symbol,
                         eventArgs.Date,
-                        close,
+                        _referencePrice,
                         _priceFactorRatio.Value
                     );
                     // let the config know about it for normalization
                     _config.SumOfDividends += baseData.Distribution;
                     _priceFactorRatio = null;
+                    _referencePrice = 0;
 
                     yield return baseData;
                 }
 
                 // check the factor file to see if we have a dividend event tomorrow
                 decimal priceFactorRatio;
-                if (_factorFile.HasDividendEventOnNextTradingDay(eventArgs.Date, out priceFactorRatio))
+                decimal referencePrice;
+                if (_factorFile.HasDividendEventOnNextTradingDay(eventArgs.Date, out priceFactorRatio, out referencePrice))
                 {
                     _priceFactorRatio = priceFactorRatio;
+                    _referencePrice = referencePrice;
                 }
             }
         }

Engine/DataFeeds/Enumerators/Factories/CorporateEventEnumeratorFactory.cs

@@ -19,6 +19,7 @@ using System.Collections.Generic;
 using System.Linq;
 using QuantConnect.Data;
 using QuantConnect.Data.Auxiliary;
+using QuantConnect.Data.Market;
 using QuantConnect.Interfaces;
 using QuantConnect.Logging;
 
@@ -86,6 +87,20 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators.Factories
             return new SynchronizingEnumerator(dataEnumerator, enumerator);
         }
 
+        /// <summary>
+        /// Centralized logic used by the data feeds to determine if we should emit auxiliary base data points.
+        /// For equities we only want to emit split/dividends events for non internal and only for <see cref="TradeBar"/> configurations
+        /// this last part is because equities also have <see cref="QuoteBar"/> subscriptions.
+        /// </summary>
+        /// <remarks>The <see cref="TimeSliceFactory"/> does not allow for multiple dividends/splits per symbol in the same time slice
+        /// but we don't want to rely only on that and make an explicit decision here.</remarks>
+        /// <remarks>History provider is never emitting auxiliary data points</remarks>
+        public static bool ShouldEmitAuxiliaryBaseData(SubscriptionDataConfig config)
+        {
+            return config.SecurityType != SecurityType.Equity || !config.IsInternalFeed
+                && (config.Type == typeof(TradeBar) || config.Type == typeof(Tick) && config.TickType == TickType.Trade);
+        }
+
         private static MapFile GetMapFileToUse(
             SubscriptionDataConfig config,
             MapFileResolver mapFileResolver)

Engine/DataFeeds/Enumerators/Factories/SubscriptionDataReaderSubscriptionEnumeratorFactory.cs

@@ -15,10 +15,10 @@
 */
 
 using System;
-using System.Collections.Concurrent;
 using System.Collections.Generic;
 using QuantConnect.Data;
 using QuantConnect.Data.Auxiliary;
+using QuantConnect.Data.Market;
 using QuantConnect.Data.UniverseSelection;
 using QuantConnect.Interfaces;
 using QuantConnect.Lean.Engine.Results;
@@ -27,9 +27,9 @@ using QuantConnect.Util;
 namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators.Factories
 {
     /// <summary>
-    /// Provides an implementation of <see cref="ISubscriptionEnumeratorFactory"/> that used the
-    /// <see cref="SubscriptionDataReader"/>
+    /// Provides an implementation of <see cref="ISubscriptionEnumeratorFactory"/> that used the <see cref="SubscriptionDataReader"/>
     /// </summary>
+    /// <remarks>Only used on backtesting by the <see cref="FileSystemDataFeed"/></remarks>
     public class SubscriptionDataReaderSubscriptionEnumeratorFactory : ISubscriptionEnumeratorFactory, IDisposable
     {
         private readonly bool _isLiveMode;
@@ -113,7 +113,7 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators.Factories
                 _factorFileProvider,
                 dataReader,
                 mapFileResolver,
-                _includeAuxiliaryData,
+                _includeAuxiliaryData && CorporateEventEnumeratorFactory.ShouldEmitAuxiliaryBaseData(request.Configuration),
                 request.StartTimeLocal,
                 _enablePriceScaling);
 

Engine/DataFeeds/Enumerators/PriceScaleFactorEnumerator.cs

@@ -19,7 +19,6 @@ using System.Collections;
 using System.Collections.Generic;
 using QuantConnect.Data;
 using QuantConnect.Data.Auxiliary;
-using QuantConnect.Data.Market;
 
 namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators
 {

Engine/DataFeeds/Enumerators/SplitEventProvider.cs

@@ -30,6 +30,7 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators
         // we set the split factor when we encounter a split in the factor file
         // and on the next trading day we use this data to produce the split instance
         private decimal? _splitFactor;
+        private decimal _referencePrice;
         private FactorFile _factorFile;
         private MapFile _mapFile;
         private SubscriptionDataConfig _config;
@@ -65,10 +66,14 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators
                 var factor = _splitFactor;
                 if (factor != null)
                 {
-                    var close = AuxiliaryDataEnumerator.GetRawClose(
-                        eventArgs.LastBaseData?.Price ?? 0,
-                        _config);
+                    var close = _referencePrice;
+                    if (close == 0)
+                    {
+                        throw new InvalidOperationException($"Zero reference price for {_config.Symbol} split at {eventArgs.Date}");
+                    }
+
                     _splitFactor = null;
+                    _referencePrice = 0;
                     yield return new Split(
                         eventArgs.Symbol,
                         eventArgs.Date,
@@ -78,15 +83,15 @@ namespace QuantConnect.Lean.Engine.DataFeeds.Enumerators
                 }
 
                 decimal splitFactor;
-                if (_factorFile.HasSplitEventOnNextTradingDay(eventArgs.Date, out splitFactor))
+                decimal referencePrice;
+                if (_factorFile.HasSplitEventOnNextTradingDay(eventArgs.Date, out splitFactor, out referencePrice))
                 {
                     _splitFactor = splitFactor;
+                    _referencePrice = referencePrice;
                     yield return new Split(
                         eventArgs.Symbol,
                         eventArgs.Date,
-                        AuxiliaryDataEnumerator.GetRawClose(
-                            eventArgs.LastBaseData?.Price ?? 0,
-                            _config),
+                        eventArgs.LastRawPrice ?? 0,
                         splitFactor,
                         SplitType.Warning);
                 }

Engine/DataFeeds/LiveTradingDataFeed.cs

@@ -239,7 +239,7 @@ namespace QuantConnect.Lean.Engine.DataFeeds
                     EventHandler handler = (sender, args) => subscription?.OnNewDataAvailable();
                     enumerator = _dataQueueHandler.Subscribe(request.Configuration, handler);
 
-                    if (request.Configuration.Symbol.SecurityType == SecurityType.Equity && !request.Configuration.IsInternalFeed)
+                    if (request.Configuration.SecurityType == SecurityType.Equity && CorporateEventEnumeratorFactory.ShouldEmitAuxiliaryBaseData(request.Configuration))
                     {
                         var dividends = _dataQueueHandler.Subscribe(new SubscriptionDataConfig(request.Configuration, typeof(Dividend)), handler);
                         var splits = _dataQueueHandler.Subscribe(new SubscriptionDataConfig(request.Configuration, typeof(Split)), handler);

Engine/DataFeeds/SubscriptionDataReader.cs

@@ -73,6 +73,7 @@ namespace QuantConnect.Lean.Engine.DataFeeds
         private readonly bool _isLiveMode;
 
         private BaseData _previous;
+        private decimal? _lastRawPrice;
         private readonly IEnumerator<DateTime> _tradeableDates;
 
         // used when emitting aux data from within while loop
@@ -424,6 +425,8 @@ namespace QuantConnect.Lean.Engine.DataFeeds
                     // we've satisfied user and market hour filters, so this data is good to go as current
                     Current = instance;
 
+                    // we keep the last raw price registered before we return so we are not affected by anyone (price scale) modifying our current
+                    _lastRawPrice = Current.Price;
                     return true;
                 }
 
@@ -571,7 +574,7 @@ namespace QuantConnect.Lean.Engine.DataFeeds
             {
                 date = _tradeableDates.Current;
 
-                OnNewTradableDate(new NewTradableDateEventArgs(date, _previous, _config.Symbol));
+                OnNewTradableDate(new NewTradableDateEventArgs(date, _previous, _config.Symbol, _lastRawPrice));
 
                 if (_pastDelistedDate || date > _delistingDate)
                 {

Engine/Setup/BrokerageSetupHandler.cs

@@ -195,13 +195,13 @@ namespace QuantConnect.Lean.Engine.Setup
                     return false;
                 }
 
-                var message = $"{brokerage.Name} account base currency: {brokerage.AccountBaseCurrency}";
+                var message = $"{brokerage.Name} account base currency: {brokerage.AccountBaseCurrency ?? algorithm.AccountCurrency}";
 
                 Log.Trace($"BrokerageSetupHandler.Setup(): {message}");
 
                 algorithm.Debug(message);
 
-                if (brokerage.AccountBaseCurrency != algorithm.AccountCurrency)
+                if (brokerage.AccountBaseCurrency != null && brokerage.AccountBaseCurrency != algorithm.AccountCurrency)
                 {
                     algorithm.SetAccountCurrency(brokerage.AccountBaseCurrency);
                 }

Engine/Storage/LocalObjectStore.cs

@@ -19,6 +19,7 @@ using System.Collections.Concurrent;
 using System.Collections.Generic;
 using System.IO;
 using System.Linq;
+using System.Text;
 using System.Threading;
 using QuantConnect.Configuration;
 using QuantConnect.Interfaces;
@@ -59,6 +60,7 @@ namespace QuantConnect.Lean.Engine.Storage
         private TimeSpan _persistenceInterval;
         private readonly string _storageRoot = Path.GetFullPath(Config.Get("object-store-root", "./storage"));
         private readonly ConcurrentDictionary<string, byte[]> _storage = new ConcurrentDictionary<string, byte[]>();
+        private readonly object _persistLock = new object();
 
         /// <summary>
         /// Provides access to the controls governing behavior of this instance, such as the persistence interval
@@ -90,14 +92,8 @@ namespace QuantConnect.Lean.Engine.Storage
 
             Controls = controls;
 
-            if (Controls.StoragePermissions.HasFlag(FileAccess.Read))
-            {
-                foreach (var file in Directory.EnumerateFiles(AlgorithmStorageRoot))
-                {
-                    var contents = File.ReadAllBytes(file);
-                    _storage[Path.GetFileName(file)] = contents;
-                }
-            }
+            // Load in any already existing objects in the storage directory
+            LoadExistingObjects();
 
             // if <= 0 we disable periodic persistence and make it synchronous
             if (Controls.PersistenceIntervalSeconds > 0)
@@ -107,6 +103,23 @@ namespace QuantConnect.Lean.Engine.Storage
             }
         }
 
+        /// <summary>
+        /// Loads objects from the AlgorithmStorageRoot into the ObjectStore
+        /// </summary>
+        protected virtual void LoadExistingObjects()
+        {
+            if (Controls.StoragePermissions.HasFlag(FileAccess.Read))
+            {
+                foreach (var file in Directory.EnumerateFiles(AlgorithmStorageRoot))
+                {
+                    // Read the contents of the file and decode the filename to get our key
+                    var contents = File.ReadAllBytes(file);
+                    var key = Base64ToKey(Path.GetFileName(file));
+                    _storage[key] = contents;
+                }
+            }
+        }
+
         /// <summary>
         /// Determines whether the store contains data for the specified key
         /// </summary>
@@ -133,23 +146,17 @@ namespace QuantConnect.Lean.Engine.Storage
         /// <returns>A byte array containing the data</returns>
         public byte[] ReadBytes(string key)
         {
-            if (key == null)
-            {
-                throw new ArgumentNullException(nameof(key));
-            }
-            if (!Controls.StoragePermissions.HasFlag(FileAccess.Read))
-            {
-                throw new InvalidOperationException($"LocalObjectStore.ReadBytes(): {NoReadPermissionsError}");
-            }
-
-            byte[] data;
-            if (!_storage.TryGetValue(key, out data))
+            // Ensure we have the key, also takes care of null or improper access
+            if (!ContainsKey(key))
             {
                 throw new KeyNotFoundException($"Object with key '{key}' was not found in the current project. " +
                     "Please use ObjectStore.ContainsKey(key) to check if an object exists before attempting to read."
                 );
             }
 
+            byte[] data;
+            _storage.TryGetValue(key, out data);
+
             return data;
         }
 
@@ -165,6 +172,10 @@ namespace QuantConnect.Lean.Engine.Storage
             {
                 throw new ArgumentNullException(nameof(key));
             }
+            if (key.Contains("?"))
+            {
+                throw new ArgumentException($"LocalObjectStore.SaveBytes(): char '?' is not supported in the key {key}");
+            }
             if (!Controls.StoragePermissions.HasFlag(FileAccess.Write))
             {
                 throw new InvalidOperationException($"LocalObjectStore.SaveBytes(): {NoWritePermissionsError}");
@@ -189,22 +200,25 @@ namespace QuantConnect.Lean.Engine.Storage
         /// </summary>
         protected bool InternalSaveBytes(string key, byte[] contents)
         {
-            var fileCount = 0;
-            var expectedStorageSizeBytes = 0L;
+            // Before saving confirm we are abiding by the control rules
+            // Start by counting our file and its length
+            var fileCount = 1;
+            var expectedStorageSizeBytes = contents.Length;
             foreach (var kvp in _storage)
             {
-                fileCount++;
-                if (string.Equals(kvp.Key, key))
+                if (key.Equals(kvp.Key))
                 {
-                    // we use the New content size
-                    expectedStorageSizeBytes += contents.Length;
+                    // Skip we have already counted this above
+                    // If this key was already in storage it will be replaced.
                 }
                 else
                 {
+                    fileCount++;
                     expectedStorageSizeBytes += kvp.Value.Length;
                 }
             }
 
+            // Verify we are within FileCount limit
             if (fileCount > Controls.StorageFileCount)
             {
                 var message = $"LocalObjectStore.InternalSaveBytes(): at file capacity: {fileCount}. Unable to save: '{key}'";
@@ -213,6 +227,7 @@ namespace QuantConnect.Lean.Engine.Storage
                 return false;
             }
 
+            // Verify we are within Storage limit
             var expectedStorageSizeMb = BytesToMb(expectedStorageSizeBytes);
             if (expectedStorageSizeMb > Controls.StorageLimitMB)
             {
@@ -222,8 +237,8 @@ namespace QuantConnect.Lean.Engine.Storage
                 return false;
             }
 
+            // Add the entry
             _storage.AddOrUpdate(key, k => contents, (k, v) => contents);
-
             return true;
         }
 
@@ -248,19 +263,6 @@ namespace QuantConnect.Lean.Engine.Storage
             {
                 _dirty = true;
 
-                try
-                {
-                    var path = GetFilePathForKey(key);
-                    if (File.Exists(path))
-                    {
-                        File.Delete(path);
-                    }
-                }
-                catch (Exception exception)
-                {
-                    Log.Error(exception);
-                }
-
                 // if <= 0 we disable periodic persistence and make it synchronous
                 if (Controls.PersistenceIntervalSeconds <= 0)
                 {
@@ -277,21 +279,21 @@ namespace QuantConnect.Lean.Engine.Storage
         /// </summary>
         /// <param name="key">The object key</param>
         /// <returns>The path for the file</returns>
-        public string GetFilePath(string key)
+        public virtual string GetFilePath(string key)
         {
-            if (key == null)
+            // Ensure we have an object for that key
+            if (!ContainsKey(key))
             {
-                throw new ArgumentNullException(nameof(key));
+                throw new KeyNotFoundException($"Object with key '{key}' was not found in the current project. " +
+                    "Please use ObjectStore.ContainsKey(key) to check if an object exists before attempting to read."
+                );
             }
 
-            // read from RAM
-            var contents = ReadBytes(key);
+            // Persist to ensure pur files are up to date
+            Persist();
 
-            // write byte array to storage directory
-            var path = GetFilePathForKey(key);
-            File.WriteAllBytes(path, contents);
-
-            return path;
+            // Fetch the path to file and return it
+            return PathForKey(key);
         }
 
         /// <summary>
@@ -311,8 +313,7 @@ namespace QuantConnect.Lean.Engine.Storage
                 }
 
                 // if the object store was not used, delete the empty storage directory created in Initialize.
-                // can be null if not initialized
-                if (AlgorithmStorageRoot != null && !Directory.EnumerateFileSystemEntries(AlgorithmStorageRoot).Any())
+                if (AlgorithmStorageRoot != null && !Directory.GetFileSystemEntries(AlgorithmStorageRoot).Any())
                 {
                     Directory.Delete(AlgorithmStorageRoot);
                 }
@@ -340,11 +341,16 @@ namespace QuantConnect.Lean.Engine.Storage
         }
 
         /// <summary>
-        /// Get's the file path for a giving object store key
+        /// Get's a file path for a given key.
+        /// Internal use only because it does not guarantee the existence of the file.
         /// </summary>
-        private string GetFilePathForKey(string key)
+        protected string PathForKey(string key)
         {
-            return Path.Combine(AlgorithmStorageRoot, $"{key.ToMD5()}.dat");
+            // We use an encoded filename because certain keys will cause problems with persisting
+            // data to a file; we use Base64 because it allow us to use all chars except '?' in our
+            // key, this is because '?' in the right place will output '/' which will break during
+            // persist
+            return Path.Combine(AlgorithmStorageRoot, $"{KeyToBase64(key)}");
         }
 
         /// <summary>
@@ -352,32 +358,35 @@ namespace QuantConnect.Lean.Engine.Storage
         /// </summary>
         private void Persist()
         {
-            if (!_dirty)
+            // Acquire the persist lock
+            lock (_persistLock)
             {
-                return;
-            }
-
-            try
-            {
-
-                // pause timer will persisting
-                _persistenceTimer?.Change(Timeout.Infinite, Timeout.Infinite);
-
-                if (PersistData(this))
+                // If there are no changes we are fine
+                if (!_dirty)
                 {
-                    _dirty = false;
+                    return;
                 }
 
-            }
-            catch (Exception err)
-            {
-                Log.Error("LocalObjectStore.Persist()", err);
-                OnErrorRaised(err);
-            }
-            finally
-            {
-                // restart timer following end of persistence
-                _persistenceTimer?.Change(_persistenceInterval, _persistenceInterval);
+                try
+                {
+                    // Pause timer while persisting
+                    _persistenceTimer?.Change(Timeout.Infinite, Timeout.Infinite);
+
+                    if (PersistData(this))
+                    {
+                        _dirty = false;
+                    }
+                }
+                catch (Exception err)
+                {
+                    Log.Error("LocalObjectStore.Persist()", err);
+                    OnErrorRaised(err);
+                }
+                finally
+                {
+                    // restart timer following end of persistence
+                    _persistenceTimer?.Change(_persistenceInterval, _persistenceInterval);
+                }
             }
         }
 
@@ -390,15 +399,27 @@ namespace QuantConnect.Lean.Engine.Storage
         {
             try
             {
+                // Delete any files that are no longer saved in the store
+                foreach (var filepath in Directory.EnumerateFiles(AlgorithmStorageRoot))
+                {
+                    var filename = Path.GetFileName(filepath);
+                    if (!_storage.ContainsKey(Base64ToKey(filename)))
+                    {
+                        File.Delete(filepath);
+                    }
+                }
+
+                // Write all our store data to disk
                 foreach (var kvp in data)
                 {
-                    var path = Path.Combine(AlgorithmStorageRoot, kvp.Key);
+                    // Get a path for this key and write to it
+                    var path = PathForKey(kvp.Key);
                     File.WriteAllBytes(path, kvp.Value);
                 }
 
                 return true;
             }
-            catch (Exception err)
+            catch (Exception err) 
             {
                 Log.Error("LocalObjectStore.PersistData()", err);
                 OnErrorRaised(err);
@@ -421,5 +442,27 @@ namespace QuantConnect.Lean.Engine.Storage
         {
             return bytes / 1024.0 / 1024.0;
         }
+
+        /// <summary>
+        /// Convert a given key to a Base64 string
+        /// </summary>
+        /// <param name="key">Key to be encoded</param>
+        /// <returns>Base64 hash string</returns>
+        private static string KeyToBase64(string key)
+        {
+            var textAsBytes = Encoding.UTF8.GetBytes(key);
+            return Convert.ToBase64String(textAsBytes);
+        }
+
+        /// <summary>
+        /// Convert a given Base64 string back to a key
+        /// </summary>
+        /// <param name="hash">Hash to be decoded</param>
+        /// <returns>Key string</returns>
+        private static string Base64ToKey(string hash)
+        {
+            var textAsBytes = Convert.FromBase64String(hash);
+            return Encoding.UTF8.GetString(textAsBytes);
+        }
     }
 }

Launcher/Python.Runtime.dll.config

@@ -1,2 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
-<configuration></configuration>
+<configuration>
+    <!-- For Linux and Mac users: uncomment and fill either of the following: -->
+    <!-- <dllmap os="linux" dll="python3.6m" target = "/home/{your_user_name}/miniconda3/envs/{qc_environment}/lib/libpython3.6m.so"/> -->
+    <!-- <dllmap os="osx" dll="python3.6m" target = "/Users/{your_user_name}/anaconda3/lib/libpython3.6m.dylib"/> -->
+</configuration>

Research/KitchenSinkCSharpQuantBookTemplate.ipynb

@@ -15,6 +15,17 @@
    "metadata": {},
    "source": [
     "## QuantBook Basics\n",
+    "The following example is ready to be used in our Docker container, reference the ReadMe for more details on setting this up.\n",
+    "\n",
+    "\n",
+    "\n",
+    "In order to use this notebook locally you will need to make a few small changes:\n",
+    "\n",
+    "1. Either create the notebook in your build folder (`bin/debug`) **or** set working directory of the notebook to it like so in the first cell:\n",
+    "\n",
+    "    ```Directory.SetCurrentDirectory(\"PathToLean/Lean/Launcher/bin/Debug/\");```\n",
+    "\n",
+    "2. Load \"QuantConnect.csx\" instead of \"../QuantConnect.csx\", this is again because of the Notebook position relative to the build files. \n",
     "\n",
     "### Start QuantBook\n",
     "- Load \"QuantConnect.csx\" with all the basic imports\n",
@@ -27,12 +38,46 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#load \"QuantConnect.csx\"\n",
-    "using QuantConnect.Data.Custom;\n",
-    "using QuantConnect.Data.Market;\n",
+    "#load \"../QuantConnect.csx\"\n",
     "var qb = new QuantBook();"
    ]
   },
+  {
+   "source": [
+    "### Using the Web API\n",
+    "Our script `QuantConnect.csx` automatically loads an instance of the web API for you to use.**\n",
+    "\n",
+    "Look at Lean's [Api](https://github.com/QuantConnect/Lean/tree/master/Api) class for more functions to interact with the cloud\n",
+    "\n",
+    "\n",
+    "##### **Note: This will only connect if you have your User ID and Api token in `config.json` "
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Show that our api object is connected to the Web Api\n",
+    "Console.WriteLine(api.Connected);"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Get our list of projects from the cloud and print their names\n",
+    "var projectResponse = api.ListProjects();\n",
+    "foreach (var project in projectResponse.Projects) {\n",
+    "    Console.WriteLine(project.Name);\n",
+    "}"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -42,16 +87,16 @@
    ]
   },
   {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "var spy = qb.AddEquity(\"SPY\");\n",
-      "var eur = qb.AddForex(\"EURUSD\");\n",
-      "var btc = qb.AddCrypto(\"BTCUSD\");\n",
-      "var fxv = qb.AddData<FxcmVolume>(\"EURUSD_Vol\", Resolution.Hour);"
-    ]
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "var spy = qb.AddEquity(\"SPY\");\n",
+    "var eur = qb.AddForex(\"EURUSD\");\n",
+    "var btc = qb.AddCrypto(\"BTCUSD\");\n",
+    "var fxv = qb.AddData<FxcmVolume>(\"EURUSD_Vol\", Resolution.Hour);"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -65,64 +110,71 @@
    ]
   },
   {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "// Gets historical data from the subscribed assets, the last 360 datapoints with daily resolution\n",
-      "var h1 = qb.History(qb.Securities.Keys, 360, Resolution.Daily);"
-    ]
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Gets historical data from the subscribed assets, the last 360 datapoints with daily resolution\n",
+    "var h1 = qb.History(qb.Securities.Keys, 360, Resolution.Daily);"
+   ]
   },
   {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "// Gets historical data from the subscribed assets, from the last 30 days with daily resolution\n",
-      "var h2 = qb.History(qb.Securities.Keys, TimeSpan.FromDays(360), Resolution.Daily);"
-    ]
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Gets historical data from the subscribed assets, from the last 30 days with daily resolution\n",
+    "var h2 = qb.History(qb.Securities.Keys, TimeSpan.FromDays(360), Resolution.Daily);"
+   ]
   },
   {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "// Gets historical data from the subscribed assets, between two dates with daily resolution\n",
-      "var h3 = qb.History(btc.Symbol, new DateTime(2014,1,1), DateTime.Now, Resolution.Daily);"
-    ]
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Gets historical data from the subscribed assets, between two dates with daily resolution\n",
+    "var h3 = qb.History(btc.Symbol, new DateTime(2014,1,1), DateTime.Now, Resolution.Daily);"
+   ]
   },
   {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "// Only fetchs historical data from a desired symbol\n",
-      "var h4 = qb.History(spy.Symbol, 360, Resolution.Daily);"
-    ]
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Only fetchs historical data from a desired symbol\n",
+    "var h4 = qb.History(spy.Symbol, 360, Resolution.Daily);"
+   ]
   },
   {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "// Only fetchs historical data from a desired symbol\n",
-      "var h5 = qb.History<QuoteBar>(eur.Symbol, TimeSpan.FromDays(360), Resolution.Daily);"
-    ]
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Only fetchs historical data from a desired symbol\n",
+    "var h5 = qb.History<QuoteBar>(eur.Symbol, TimeSpan.FromDays(360), Resolution.Daily);"
+   ]
   },
   {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "// Fetchs custom data\n",
-      "var h6 = qb.History<FxcmVolume>(fxv.Symbol, TimeSpan.FromDays(360));"
-    ]
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "// Fetchs custom data\n",
+    "var h6 = qb.History<FxcmVolume>(fxv.Symbol, TimeSpan.FromDays(360));"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -146,4 +198,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
+}

Research/KitchenSinkQuantBookTemplate.ipynb

@@ -17,6 +17,19 @@
    },
    "source": [
     "## QuantBook Basics\n",
+    "The following example is ready to be used in our Docker container, reference the ReadMe for more details on setting this up.\n",
+    "\n",
+    "\n",
+    "\n",
+    "In order to use this notebook locally you will need to make a few small changes:\n",
+    "\n",
+    "1. Either create the notebook in your build folder (`bin/debug`) **or** set working directory of the notebook to it like so in the first cell:\n",
+    "\n",
+    "    ```%cd \"PathToLean/Lean/Launcher/bin/Debug/```\n",
+    "\n",
+    "2. Run the following command in another cell to load in QuantConnect libraries:\n",
+    "\n",
+    "    ```%run start.py```\n",
     "\n",
     "### Start QuantBook\n",
     "- Add the references and imports\n",
@@ -29,27 +42,55 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%matplotlib inline\n",
-    "# Imports\n",
-    "from clr import AddReference\n",
-    "AddReference(\"System\")\n",
-    "AddReference(\"QuantConnect.Common\")\n",
-    "AddReference(\"QuantConnect.Jupyter\")\n",
-    "AddReference(\"QuantConnect.Indicators\")\n",
-    "from System import *\n",
-    "from QuantConnect import *\n",
-    "from QuantConnect.Data.Custom import *\n",
-    "from QuantConnect.Data.Market import TradeBar, QuoteBar\n",
-    "from QuantConnect.Jupyter import *\n",
-    "from QuantConnect.Indicators import *\n",
-    "from datetime import datetime, timedelta\n",
-    "import matplotlib.pyplot as plt\n",
-    "import pandas as pd\n",
-    "\n",
-    "# Create an instance\n",
+    "#Import any needed libraries\n",
+    "from datetime import *"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create an instance of our QuantBook\n",
     "qb = QuantBook()"
    ]
   },
+  {
+   "source": [
+    "### Using the Web API\n",
+    "Our script `start.py` automatically loads an instance of the web API for you to use.**\n",
+    "\n",
+    "Look at Lean's [Api](https://github.com/QuantConnect/Lean/tree/master/Api) class for more functions to interact with the cloud\n",
+    "\n",
+    "\n",
+    "##### **Note: This will only connect if you have your User ID and Api token in `config.json` \n"
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Show that our api object is connected to the Web Api\n",
+    "print(api.Connected)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get our list of projects from the cloud and print their names\n",
+    "projectResponse = api.ListProjects()\n",
+    "for project in projectResponse.Projects:\n",
+    "    print(project.Name)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {
@@ -68,8 +109,7 @@
    "source": [
     "spy = qb.AddEquity(\"SPY\")\n",
     "eur = qb.AddForex(\"EURUSD\")\n",
-    "btc = qb.AddCrypto(\"BTCUSD\")\n",
-    "fxv = qb.AddData[FxcmVolume](\"EURUSD_Vol\", Resolution.Hour)"
+    "btc = qb.AddCrypto(\"BTCUSD\")"
    ]
   },
   {
@@ -415,9 +455,13 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
+   "name": "Python 3.6.8 64-bit",
+   "display_name": "Python 3.6.8 64-bit",
+   "metadata": {
+    "interpreter": {
+     "hash": "78516bfd76dd3216407ec6a939363bc6db05a52b853229e9b8ca6f74d4ab93c2"
+    }
+   }
   },
   "language_info": {
    "codemirror_mode": {
@@ -429,9 +473,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.4"
+   "version": "3.6.8-final"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
+}

Research/QuantBook.cs

@@ -95,8 +95,21 @@ namespace QuantConnect.Research
                     _pandas = Py.Import("pandas");
                 }
 
-                // By default, set start date to end data which is yesterday
-                SetStartDate(EndDate);
+                // Issue #4892 : Set start time relative to NY time 
+                // when the data is available from the previous day
+                var newYorkTime = DateTime.UtcNow.ConvertFromUtc(TimeZones.NewYork);
+                var hourThreshold = Config.GetInt("qb-data-hour", 9);
+                
+                // If it is after our hour threshold; then we can use today 
+                if (newYorkTime.Hour >= hourThreshold)
+                {
+                    SetStartDate(newYorkTime);
+                }
+                else
+                {
+                    SetStartDate(newYorkTime - TimeSpan.FromDays(1));
+                }
+                
 
                 // Sets PandasConverter
                 SetPandasConverter();

Research/QuantConnect.Research.csproj

@@ -170,6 +170,9 @@
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </Content>
     <None Include="readme.md" />
+    <Content Include="start.py">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
   </ItemGroup>
   <ItemGroup>
     <Analyzer Include="..\packages\Microsoft.CodeAnalysis.VersionCheckAnalyzer.2.9.3\analyzers\dotnet\Microsoft.CodeAnalysis.VersionCheckAnalyzer.dll" />

Research/QuantConnect.csx

@@ -17,6 +17,7 @@
 #r "QuantConnect.Configuration.dll"
 #r "QuantConnect.Lean.Engine.dll"
 #r "QuantConnect.Algorithm.CSharp.dll"
+#r "QuantConnect.Api.dll"
 // Note: #r directives must be in the beggining of the file
 
 /*
@@ -54,6 +55,7 @@ using QuantConnect.Algorithm.Framework.Alphas;
 using QuantConnect.Algorithm.Framework.Portfolio;
 using QuantConnect.Algorithm.Framework.Execution;
 using QuantConnect.Algorithm.Framework.Risk;
+using QuantConnect.Api;
 using QuantConnect.Parameters;
 using QuantConnect.Benchmarks;
 using QuantConnect.Brokerages;
@@ -76,4 +78,11 @@ using QuantConnect.Scheduling;
 using QuantConnect.Securities;
 using QuantConnect.Securities.Equity;
 using QuantConnect.Securities.Forex;
-using QuantConnect.Securities.Interfaces;
+using QuantConnect.Securities.Interfaces;
+using QuantConnect.Configuration;
+
+// Loads up a connection to our API for use in the research environment
+Api api = new Api();
+api.Initialize(Config.GetInt("job-user-id", 1), 
+    Config.Get("api-access-token", "default"),
+    Config.Get("data-folder"));

Research/docker.cfg

@@ -1,3 +1,4 @@
-image=
-data_dir=
-notebook_dir=
+IMAGE=quantconnect/research:latest
+DATA_DIR=
+NOTEBOOK_DIR=
+UPDATE=Y

Research/readme.md

@@ -8,6 +8,8 @@ The up to date docker image is available at [quantconnect/research](https://hub.
 
 
 # Using the Docker Image
+
+## Starting the Container
 The docker image we created can be started using the included .bat/.sh file in this directory (Lean/Research). These scripts take care of all the work required to get the notebook container setup and started for use. Including launching a browser to the notebook lab environment for you.
 
 From a terminal launch the run_docker_notebook.bat/.sh script; there are a few options on how to launch this:
@@ -19,30 +21,62 @@ From a terminal launch the run_docker_notebook.bat/.sh script; there are a few o
 
  2. Using the **docker.cfg** to store args for repeated use; any blank entries will resort to default values! ex: `./run_docker_notebook.bat docker.cfg`
   
-        image=
-        data_dir=
-        notebook_dir=
+         IMAGE=quantconnect/research:latest
+         DATA_DIR=
+         NOTEBOOK_DIR=
 
- 3. Inline arguments; anything you don't enter will use the default args! ex: `./run_docker.bat image=quantconnect/research:latest`
+ 3. Inline arguments; anything you don't enter will use the default args! ex: `./run_docker.bat IMAGE=quantconnect/research:latest`
       *    Accepted args for inline include all listed in the file **docker.cfg**
 
-Once the docker image starts, the script will attempt to open your browser to the Jupyter notebook web app, if this fails go to `localhost:8888`
+Once the docker image starts, the script will attempt to open your browser to the Jupyter notebook web app, if this fails open your browser and go to `localhost:8888`
 
-When you are done with the research environment be sure to stop the container with either Docker's dashboard or through the CLI.
+<br>
 
-## Note for C#
-When using C# for research notebooks it requires that you load our CSX file `QuantConnect.csx` into your notebook. In this setup, the file is one directory above the notebooks default dir. Be sure to use the following line to load in this csx file:
+## C# Notebook
+When using C# for research notebooks it requires that you load our setup script CSX file `QuantConnect.csx` into your notebook. This will load our QuantConnect libraries into your C# Kernel. In this setup, the file is one directory above the notebooks dir. Be sure to use the following line in your first cell to load in this csx file:
 
 `load "../QuantConnect.csx"`
 
+After this the environment is ready to use; take a look at our reference notebook `KitchenSinkCSharpQuantBookTemplate.ipynb` for an example of how to use our `QuantBook` interface!
+
+<br>
+
+## Python Notebook
+With Python we have a setup script that will automatically load QuantBooks libraries into the Python kernel so there is no need to import them. 
+
+You notebook is ready to use; take a look at our reference notebook `KitchenSinkQuantBookTemplate.ipynb` for an example of how to use our `QuantBook` interface!
+
+<br>
+
+## Using the Web Api from Notebook
+Both of our setup scripts for Python & C# include a instantiated `Api` object under the variable name `api`. Before you can use this api object to interact with the cloud you must edit your config in the **root** of your Notebook directory. Once this has been done once, it does not need to be done again.
+
+In `config.json` add the following entries with your respective values
+```
+job-user-id: 12345, // Your id here
+api-access-token: "token13432", // Your api token here
+```
+
+Once this has been done, you may restart your kernel and begin to use the `api` variable. 
+Reference our examples mentioned above for practical uses of this object.
+
+<br>
+
+## Shutting Down the Notebook Lab
+When you are done with the research environment be sure to stop the container with either **Docker's dashboard** or through the **Docker CLI** with `docker kill LeanResearch`.
+
+<br>
+
+
 ## Build a new image
 For most users this will not be necessary, simply use `docker pull quantconnect/research` to get the latest image.
 
-
 `docker build -t quantconnect/research - < DockerfileJupyter` will build a new docker image using the latest version of lean. To build from particular tag of lean a build arg can be provided, for example `--build-arg LEAN_TAG=8631`.
 
+<br>
 
-# Running Jupyter Locally
+# Running Jupyter Locally 
+Note: we recommend using the above approach with our Docker container, where the setup and evironment is tested and stable.
 
 Before we enable Jupyter support, follow [Lean installation](https://github.com/QuantConnect/Lean#installation-instructions)
 and [Python installation](https://github.com/QuantConnect/Lean/tree/master/Algorithm.Python#quantconnect-python-algorithm-project) to get LEAN running Python algorithms in your machine.