Version 0.2.64

sync dev -> main

.github/workflows/auto_close_default_issues.yml

@@ -0,0 +1,36 @@
+name: Auto-close issues using default template
+on:
+  issues:
+    types: [opened]
+
+jobs:
+  check-template:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check if issue uses custom template
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const issue = context.payload.issue;
+            const body = issue.body || '';
+            
+            // Check for specific fields from your custom form
+            // Adjust these patterns based on your form structure
+            const hasCustomFields = body.includes('### Describe bug') || 
+                                   body.includes('### Simple code that reproduces');
+            
+            if (!hasCustomFields) {
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: issue.number,
+                body: 'This issue appears to use the default template. Stop that. Use our custom bug report form.'
+              });
+              
+              await github.rest.issues.update({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: issue.number,
+                state: 'closed'
+              });
+            }

.gitignore

@@ -9,7 +9,6 @@ yfinance.egg-info
 build/
 *.html
 *.css
-*.png
 test.ipynb
 
 # Environments
@@ -24,4 +23,4 @@ ENV/
 /doc/_build/
 /doc/source/reference/api
 !yfinance.css
-!/doc/source/development/assets/branches.png
+!/doc/source/development/assets/branches.png

CHANGELOG.rst

@@ -1,6 +1,13 @@
 Change Log
 ===========
 
+0.2.64
+------
+Prices:
+- handle dividends with FX, convert if repair=True  #2549
+- fix 'period' arg when start or end set  #2550
+earnings_dates: handle 'Event Type' properly  #2555
+
 0.2.63
 ------
 Fix download(ISIN)  # 2531

doc/source/advanced/index.rst

@@ -8,4 +8,5 @@ Advanced
    logging
    config
    caching
-   multi_level_columns
+   multi_level_columns
+   price_repair

doc/source/advanced/price_repair.rst

@@ -0,0 +1,277 @@
+************
+Price Repair
+************
+
+The new argument ``repair=True`` in ``history()`` and ``download()`` will attempt to fix a variety of price errors caused by Yahoo. Only US market data appears perfect, I guess Yahoo doesn't care much about rest of world?
+
+The returned table will have a new column ``Repaired?`` that specifies if row was repaired.
+
+Price repair
+============
+
+Missing dividend adjustment
+---------------------------
+
+If dividend in data but preceding ``Adj Close`` = ``Close``, then manually apply dividend-adjustment to ``Adj Close``.
+Note: ``Repaired?`` is NOT set to ``True`` because fix only changes ``Adj Close``
+
+.. figure:: /_static/images/repair-prices-missing-div-adjust.png
+   :alt: 8TRA.DE: repair missing dividend adjustment
+   :width: 80%
+   :align: left
+
+   8TRA.DE
+
+.. container:: clearer
+
+   ..
+
+Missing split adjustment
+------------------------
+
+If stock split in data but preceding price data is not adjusted, then manually apply stock split.
+Requires date range include 1 day after stock split for calibration - sometimes Yahoo fails to adjust prices on stock split day.
+
+.. figure:: /_static/images/repair-prices-missing-split-adjust.png
+   :alt: MOB.ST: repair missing split adjustment
+   :width: 80%
+   :align: left
+
+   MOB.ST
+
+.. container:: clearer
+
+   ..
+
+Missing data
+------------
+
+If price data is clearly missing or corrupt, then reconstructed using smaller interval e.g. ``1h`` to fix ``1d`` data.
+
+.. figure:: /_static/images/repair-prices-missing-row.png
+   :alt: 1COV.DE: repair missing row
+   :width: 80%
+   :align: left
+
+   1COV.DE missing row
+
+.. container:: clearer
+
+   ..
+
+.. figure:: /_static/images/repair-prices-missing-volume-intraday.png
+   :alt: 1COV.DE: repair missing Volume, but intraday price changed
+   :width: 80%
+   :align: left
+
+   1COV.DE missing Volume, but intraday price changed
+
+.. container:: clearer
+
+   ..
+
+.. figure:: /_static/images/repair-prices-missing-volume-daily.png
+   :alt: 0316.HK: repair missing Volume, but daily price changed
+   :width: 80%
+   :align: left
+
+   0316.HK missing Volume, but daily price changed
+
+.. container:: clearer
+
+   ..
+
+100x errors
+-----------
+
+Sometimes Yahoo mixes up currencies e.g. $/cents or £/pence. So some prices are 100x wrong.
+Sometimes they are spread randomly through data - these detected with ``scipy`` module.
+Other times they are in a block, because Yahoo decided one day to permanently switch currency.
+
+.. figure:: /_static/images/repair-prices-100x.png
+   :alt: AET.L: repair 100x
+   :width: 80%
+   :align: left
+
+   AET.L
+
+Price reconstruction - algorithm notes
+--------------------------------------
+
+Spam minimised by grouping fetches. Tries to be aware of data limits e.g. ``1h`` cannot be fetched beyond 2 years.
+
+If Yahoo eventually does fix the bad data that required reconstruction, you will see it's slightly different to reconstructed prices and volume often significantly different. Best I can do, and beats missing data.
+
+Dividend repair (new)
+=====================
+
+Fix errors in dividends:
+
+1. adjustment missing or 100x too small/big for the dividend
+2. duplicate dividend (within 7 days)
+3. dividend 100x too big/small for the ex-dividend price drop
+4. ex-div date wrong (price drop is few days/weeks after)
+
+Most errors I've seen are on London stock exchange (£/pence mixup), but no exchange is safe.
+
+IMPORTANT - false positives
+---------------------------
+
+Because fixing (3) relies on price action, there is a chance of a "false positive" (FP) - thinking an error exists when data is good.
+FP rate increases with longer intervals, so only 1d intervals are repaired. If you request repair on multiday intervals (weekly etc), then: 1d is fetched from Yahoo, repaired, then resampled - **this has nice side-effect of solving Yahoo's flawed way of div-adjusting multiday intervals.**
+
+FP rate on 1d is tiny. They tend to happen with tiny dividends e.g. 0.5%, mistaking normal price volatility for an ex-div drop 100x bigger than the dividend, causing repair of the "too small" dividend (repair logic already tries to account for normal volatility by subtracting median). Either accept the risk, or fetch 6-12 months of prices with at least 2 dividends - then can analyse the dividends together to identify false positives.
+
+Adjustment missing
+------------------
+
+1398.HK
+
+.. code-block:: text
+
+   # ORIGINAL:
+                              Close  Adj Close  Dividends
+   2024-07-08 00:00:00+08:00   4.33       4.33   0.335715
+   2024-07-04 00:00:00+08:00   4.83       4.83   0.000000
+
+.. code-block:: text
+
+   # REPAIRED:
+                              Close  Adj Close  Dividends
+   2024-07-08 00:00:00+08:00   4.33   4.330000   0.335715
+   2024-07-04 00:00:00+08:00   4.83   4.494285   0.000000
+
+Adjustment too small
+--------------------
+
+3IN.L
+
+.. code-block:: text
+
+   # ORIGINAL:
+                              Close  Adj Close  Dividends
+   2024-06-13 00:00:00+01:00  3.185   3.185000    0.05950
+   2024-06-12 00:00:00+01:00  3.270   3.269405    0.00000
+
+.. code-block:: text
+
+   # REPAIRED:
+                              Close  Adj Close  Dividends
+   2024-06-13 00:00:00+01:00  3.185   3.185000    0.05950
+   2024-06-12 00:00:00+01:00  3.270   3.210500    0.00000
+
+Duplicate (within 7 days)
+-------------------------
+
+ALC.SW
+
+.. code-block:: text
+
+   # ORIGINAL:
+                                  Close  Adj Close  Dividends
+   2023-05-10 00:00:00+02:00  70.580002  70.352142       0.21
+   2023-05-09 00:00:00+02:00  65.739998  65.318443       0.21
+   2023-05-08 00:00:00+02:00  66.379997  65.745682       0.00
+
+.. code-block:: text
+
+   # REPAIRED:
+                                  Close  Adj Close  Dividends
+   2023-05-10 00:00:00+02:00  70.580002  70.352142       0.00
+   2023-05-09 00:00:00+02:00  65.739998  65.527764       0.21
+   2023-05-08 00:00:00+02:00  66.379997  65.956371       0.00
+
+Dividend too big
+----------------
+
+HLCL.L
+
+.. code-block:: text
+
+   # ORIGINAL:
+                              Close  Adj Close  Dividends
+   2024-06-27 00:00:00+01:00  2.360     2.3600       1.78
+   2024-06-26 00:00:00+01:00  2.375     2.3572       0.00
+
+   # REPAIRED:
+                              Close  Adj Close  Dividends
+   2024-06-27 00:00:00+01:00  2.360     2.3600     0.0178
+   2024-06-26 00:00:00+01:00  2.375     2.3572     0.0000
+
+Dividend & adjust too big
+-------------------------
+
+LTI.L
+
+.. code-block:: text
+
+   # ORIGINAL:
+                              Close  Adj Close     Adj  Dividends
+   2024-08-08 00:00:00+01:00  768.0      768.0  1.0000     5150.0
+   2024-08-07 00:00:00+01:00  819.0    -4331.0 -5.2882        0.0
+                              Close  Adj Close     Adj  Dividends
+   2024-08-08 00:00:00+01:00  768.0      768.0  1.0000       51.5
+   2024-08-07 00:00:00+01:00  819.0      767.5  0.9371        0.0
+
+Dividend too small
+------------------
+
+BVT.L
+
+.. code-block:: text
+
+   # ORIGINAL:
+                               Close  Adj Close     Adj  Dividends
+   2022-02-03 00:00:00+00:00  0.7534   0.675197  0.8962    0.00001
+   2022-02-01 00:00:00+00:00  0.7844   0.702970  0.8962    0.00000
+
+.. code-block:: text
+
+   # REPAIRED:
+                               Close  Adj Close     Adj  Dividends
+   2022-02-03 00:00:00+00:00  0.7534   0.675197  0.8962      0.001
+   2022-02-01 00:00:00+00:00  0.7844   0.702075  0.8950      0.000
+
+Adjusted 2x on day before
+-------------------------
+
+clue: Close < Low
+
+2020.OL
+
+.. code-block:: text
+
+   # ORIGINAL:
+                                     Low       Close   Adj Close  Dividends
+   2023-12-21 00:00:00+01:00  120.199997  121.099998  118.868782       0.18
+   2023-12-20 00:00:00+01:00  122.000000  121.900002  119.477371       0.00
+
+.. code-block:: text
+
+   # REPAIRED:
+                                     Low       Close   Adj Close  Dividends
+   2023-12-21 00:00:00+01:00  120.199997  121.099998  118.868782       0.18
+   2023-12-20 00:00:00+01:00  122.000000  122.080002  119.654045       0.00
+
+ex-div date wrong
+-----------------
+
+TETY.ST
+
+.. code-block:: text
+
+   # ORIGINAL:
+                                  Close  Adj Close  Dividends
+   2022-06-22 00:00:00+02:00  66.699997  60.085415        0.0
+   2022-06-21 00:00:00+02:00  71.599998  64.499489        0.0
+   2022-06-20 00:00:00+02:00  71.800003  64.679657        5.0
+   2022-06-17 00:00:00+02:00  71.000000  59.454838        0.0
+
+.. code-block:: text
+
+   # REPAIRED:
+                                  Close  Adj Close  Dividends
+   2022-06-22 00:00:00+02:00  66.699997  60.085415        5.0
+   2022-06-21 00:00:00+02:00  71.599998  60.007881        0.0
+   2022-06-20 00:00:00+02:00  71.800003  60.175503        0.0
+   2022-06-17 00:00:00+02:00  71.000000  59.505021        0.0

doc/source/reference/yfinance.stock.rst

@@ -20,6 +20,9 @@ Ticker stock methods
    :meth:`yfinance.scrapers.history.PriceHistory.history`
       Documentation for history
 
+   :doc:`../advanced/price_repair`
+      Documentation for price repair
+
 .. autosummary::
    :toctree: api/
    :recursive:

meta.yaml

@@ -1,5 +1,5 @@
 {% set name = "yfinance" %}
-{% set version = "0.2.63" %}
+{% set version = "0.2.64" %}
 
 package:
   name: "{{ name|lower }}"

yfinance/base.py

@@ -735,17 +735,11 @@ class TickerBase:
         params = {"lang": "en-US", "region": "US"}
         body = {
             "size": clamped_limit,
-            "query": {
-                "operator": "and",
-                "operands": [
-                    {"operator": "eq", "operands": ["ticker", self.ticker]},
-                    {"operator": "eq", "operands": ["eventtype", "2"]}
-                ]
-            },
+            "query": { "operator": "eq", "operands": ["ticker", self.ticker] },
             "sortField": "startdatetime",
             "sortType": "DESC",
             "entityIdType": "earnings",
-            "includeFields": ["startdatetime", "timeZoneShortName", "epsestimate", "epsactual", "epssurprisepct"]
+            "includeFields": ["startdatetime", "timeZoneShortName", "epsestimate", "epsactual", "epssurprisepct", "eventtype"]
         }
         response = self._data.post(url, params=params, body=body)
         json_data = response.json()
@@ -761,6 +755,14 @@ class TickerBase:
             logger.error(f'{self.ticker}: {err_msg}')
             return None
 
+        # Convert eventtype
+        # - 1 = earnings call (manually confirmed)
+        # - 2 = earnings report
+        # - 11 = stockholders meeting (manually confirmed)
+        df['Event Type'] = df['Event Type'].replace('^1$', 'Call', regex=True)
+        df['Event Type'] = df['Event Type'].replace('^2$', 'Earnings', regex=True)
+        df['Event Type'] = df['Event Type'].replace('^11$', 'Meeting', regex=True)
+
         # Calculate earnings date
         df['Earnings Date'] = pd.to_datetime(df['Event Start Date'])
         tz = self._get_ticker_tz(timeout=30)

yfinance/exceptions.py

@@ -45,7 +45,7 @@ class YFInvalidPeriodError(YFException):
         self.invalid_period = invalid_period
         self.valid_ranges = valid_ranges
         super().__init__(f"{self.ticker}: Period '{invalid_period}' is invalid, "
-                         f"must be of the format {valid_ranges}, etc.")
+                         f"must be one of: {valid_ranges}")
 
 
 class YFRateLimitError(YFException):

yfinance/multi.py

@@ -39,7 +39,7 @@ from .const import _SENTINEL_
 @utils.log_indent_decorator
 def download(tickers, start=None, end=None, actions=False, threads=True,
              ignore_tz=None, group_by='column', auto_adjust=None, back_adjust=False,
-             repair=False, keepna=False, progress=True, period="max", interval="1d",
+             repair=False, keepna=False, progress=True, period=None, interval="1d",
              prepost=False, proxy=_SENTINEL_, rounding=False, timeout=10, session=None,
              multi_level_index=True) -> Union[_pd.DataFrame, None]:
     """
@@ -49,6 +49,7 @@ def download(tickers, start=None, end=None, actions=False, threads=True,
             List of tickers to download
         period : str
             Valid periods: 1d,5d,1mo,3mo,6mo,1y,2y,5y,10y,ytd,max
+            Default: 1mo
             Either Use period parameter or use start and end
         interval : str
             Valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo

yfinance/scrapers/history.py

@@ -31,7 +31,7 @@ class PriceHistory:
         self._reconstruct_start_interval = None
 
     @utils.log_indent_decorator
-    def history(self, period="1mo", interval="1d",
+    def history(self, period=None, interval="1d",
                 start=None, end=None, prepost=False, actions=True,
                 auto_adjust=True, back_adjust=False, repair=False, keepna=False,
                 proxy=_SENTINEL_, rounding=False, timeout=10,
@@ -40,6 +40,7 @@ class PriceHistory:
         :Parameters:
             period : str
                 Valid periods: 1d,5d,1mo,3mo,6mo,1y,2y,5y,10y,ytd,max
+                Default: 1mo
                 Either Use period parameter or use start and end
             interval : str
                 Valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
@@ -60,8 +61,9 @@ class PriceHistory:
             back_adjust: bool
                 Back-adjusted data to mimic true historical prices
             repair: bool
-                Detect currency unit 100x mixups and attempt repair.
-                Default is False
+                Fixes price errors in Yahoo data: 100x, missing, bad dividend adjust.
+                Default is False.
+                Full details at: :doc:`../advanced/price_repair`.
             keepna: bool
                 Keep NaN rows returned by Yahoo?
                 Default is False
@@ -113,7 +115,7 @@ class PriceHistory:
 
         start_user = start
         end_user = end
-        if start or period is None or period.lower() == "max":
+        if start or end or (period and period.lower() == "max"):
             # Check can get TZ. Fail => probably delisted
             tz = self.tz
             if tz is None:
@@ -128,22 +130,33 @@ class PriceHistory:
                     logger.error(err_msg)
                 return utils.empty_df()
 
-            if end is None:
-                end = int(_time.time())
+        if start:
+            start = utils._parse_user_dt(start, tz)
+        if end:
+            end = utils._parse_user_dt(end, tz)
+
+        if period is None:
+            if not (start or end):
+                period = '1mo'  # default
+            elif not start:
+                # set start = end - period
+                start = int((pd.Timestamp(end, unit='s') - utils._interval_to_timedelta('1mo')).timestamp())  # -1mo
+            elif not end:
+                # set end = start + period
+                end = int((pd.Timestamp(start, unit='s') + utils._interval_to_timedelta('1mo')).timestamp())  # +1mo
+        elif period and period.lower() == "max":
+            end = int(_time.time())
+            if interval == "1m":
+                start = end - 691200  # 8 days
+            elif interval in ("2m", "5m", "15m", "30m", "90m"):
+                start = end - 5184000  # 60 days
+            elif interval in ("1h", "60m"):
+                start = end - 63072000  # 730 days
             else:
-                end = utils._parse_user_dt(end, tz)
-            if start is None:
-                if interval == "1m":
-                    start = end - 691200  # 8 days
-                elif interval in ("2m", "5m", "15m", "30m", "90m"):
-                    start = end - 5184000  # 60 days
-                elif interval in ("1h", "60m"):
-                    start = end - 63072000  # 730 days
-                else:
-                    start = end - 3122064000  # 99 years
-                start += 5 # allow for processing time
-            else:
-                start = utils._parse_user_dt(start, tz)
+                start = end - 3122064000  # 99 years
+            start += 5 # allow for processing time
+
+        if start or end:
             params = {"period1": start, "period2": end}
         else:
             period = period.lower()
@@ -328,6 +341,24 @@ class PriceHistory:
             splits = utils.set_df_tz(splits, interval, tz_exchange)
         if dividends is not None:
             dividends = utils.set_df_tz(dividends, interval, tz_exchange)
+            if 'currency' in dividends.columns:
+                # Rare, only seen with Vietnam market
+                price_currency = self._history_metadata['currency']
+                if price_currency is None:
+                    price_currency = ''
+                f_currency_mismatch = dividends['currency'] != price_currency
+                if f_currency_mismatch.any():
+                    if not repair or price_currency == '':
+                        # Append currencies to values, let user decide action.
+                        dividends['Dividends'] = dividends['Dividends'].astype(str) + ' ' + dividends['currency']
+                    else:
+                        # Attempt repair = currency conversion
+                        dividends = self._dividends_convert_fx(dividends, price_currency, repair)
+                        if (dividends['currency'] != price_currency).any():
+                            # FX conversion failed
+                            dividends['Dividends'] = dividends['Dividends'].astype(str) + ' ' + dividends['currency']
+                dividends = dividends.drop('currency', axis=1)
+
         if capital_gains is not None:
             capital_gains = utils.set_df_tz(capital_gains, interval, tz_exchange)
         if start is not None:
@@ -1019,6 +1050,45 @@ class PriceHistory:
 
         return df, currency2
 
+    def _dividends_convert_fx(self, dividends, fx, repair=False):
+        bad_div_currencies = [c for c in dividends['currency'].unique() if c != fx]
+        major_currencies = ['USD', 'JPY', 'EUR', 'CNY', 'GBP', 'CAD']
+        for c in bad_div_currencies:
+            fx2_tkr = None
+            if c == 'USD':
+                # Simple convert from USD to target FX
+                fx_tkr = f'{fx}=X'
+                reverse = False
+            elif fx == 'USD':
+                # Use same USD FX but reversed
+                fx_tkr = f'{fx}=X'
+                reverse = True
+            elif c in major_currencies and fx in major_currencies:
+                # Simple convert
+                fx_tkr = f'{c}{fx}=X'
+                reverse = False
+            else:
+                # No guarantee that Yahoo has direct FX conversion, so
+                # convert via USD
+                # - step 1: -> USD
+                fx_tkr = f'{c}=X'
+                reverse = True
+                # - step 2: USD -> FX
+                fx2_tkr = f'{fx}=X'
+
+            fx_dat = PriceHistory(self._data, fx_tkr, self.session)
+            fx_rate = fx_dat.history(period='1mo', repair=repair)['Close'].iloc[-1]
+            if reverse:
+                fx_rate = 1/fx_rate
+            dividends.loc[dividends['currency']==c, 'Dividends'] *= fx_rate
+            if fx2_tkr is not None:
+                fx2_dat = PriceHistory(self._data, fx2_tkr, self.session)
+                fx2_rate = fx2_dat.history(period='1mo', repair=repair)['Close'].iloc[-1]
+                dividends.loc[dividends['currency']==c, 'Dividends'] *= fx2_rate
+
+        dividends['currency'] = fx
+        return dividends
+
     @utils.log_indent_decorator
     def _fix_unit_mixups(self, df, interval, tz_exchange, prepost):
         if df.empty:

yfinance/utils.py

@@ -519,7 +519,10 @@ def parse_actions(data):
             dividends.set_index("date", inplace=True)
             dividends.index = _pd.to_datetime(dividends.index, unit="s")
             dividends.sort_index(inplace=True)
-            dividends.columns = ["Dividends"]
+            if 'currency' in dividends.columns and (dividends['currency'] == '').all():
+                # Currency column useless, drop it.
+                dividends = dividends.drop('currency', axis=1)
+            dividends = dividends.rename(columns={'amount': 'Dividends'})
 
         if "capitalGains" in data["events"] and len(data["events"]['capitalGains']) > 0:
             capital_gains = _pd.DataFrame(

yfinance/version.py

@@ -1 +1 @@
-version = "0.2.63"
+version = "0.2.64"