Bump version to 0.2.12

Disable annoying 'backup decrypt' msg

.github/ISSUE_TEMPLATE/bug_report.md

@@ -7,7 +7,9 @@ assignees: ''
 
 ---
 
-# READ BEFORE POSTING
+# IMPORTANT
+
+If you want help, you got to read this first, follow the instructions.
 
 ### Are you up-to-date?
 
@@ -23,20 +25,19 @@ and comparing against [PIP](https://pypi.org/project/yfinance/#history).
 
 ### Does Yahoo actually have the data?
 
-Are spelling ticker *exactly* same as Yahoo?
+Are you spelling ticker *exactly* same as Yahoo?
 
-Visit `finance.yahoo.com` and confim they have your data. Maybe your ticker was delisted.
+Then visit `finance.yahoo.com` and confirm they have the data you want. Maybe your ticker was delisted, or your expectations of `yfinance` are wrong.
 
 ### Are you spamming Yahoo?
 
-Yahoo Finance free service has limit on query rate dependent on request - roughly 500/minute for prices, 10/minute for info. Them delaying or blocking your spam is not a bug.
+Yahoo Finance free service has rate-limiting depending on request type - roughly 60/minute for prices, 10/minute for info. Once limit hit, Yahoo can delay, block, or return bad data. Not a `yfinance` bug.
 
 ### Still think it's a bug?
 
-Delete this default message and submit your bug report here, providing the following as best you can:
+Delete this default message (all of it) and submit your bug report here, providing the following as best you can:
 
-- Simple code that reproduces your problem
-- Error message, with traceback if shown
-- Info about your system:
-  - yfinance version
-  - operating system
+- Simple code that reproduces your problem, that we can copy-paste-run
+- Exception message with full traceback, or proof `yfinance` returning bad data
+- `yfinance` version and Python version
+- Operating system type

CHANGELOG.rst

@@ -1,6 +1,29 @@
 Change Log
 ===========
 
+0.2.12
+------
+Disable annoying 'backup decrypt' msg
+
+0.2.11
+------
+Fix history_metadata accesses for unusual symbols #1411
+
+0.2.10
+------
+General
+- allow using sqlite3 < 3.8.2 #1380
+- add another backup decrypt option #1379
+Prices
+- restore original download() timezone handling #1385
+- fix & improve price repair #1289 2a2928b 86d6acc
+- drop intraday intervals if in post-market but prepost=False #1311
+Info
+- fast_info improvements:
+  - add camelCase keys, add dict functions values() & items() #1368
+  - fix fast_info["previousClose"] #1383
+- catch TypeError Exception #1397
+
 0.2.9
 -----
 - Fix fast_info bugs #1362

README.md

@@ -154,19 +154,6 @@ msft.option_chain(..., proxy="PROXY_SERVER")
 ...
 ```
 
-To use a custom `requests` session (for example to cache calls to the
-API or customize the `User-agent` header), pass a `session=` argument to
-the Ticker constructor.
-
-```python
-import requests_cache
-session = requests_cache.CachedSession('yfinance.cache')
-session.headers['User-agent'] = 'my-program/1.0'
-ticker = yf.Ticker('msft', session=session)
-# The scraped response will be stored in the cache
-ticker.actions
-```
-
 To initialize multiple `Ticker` objects, use
 
 ```python
@@ -187,62 +174,46 @@ import yfinance as yf
 data = yf.download("SPY AAPL", start="2017-01-01", end="2017-04-30")
 ```
 
-I've also added some options to make life easier :)
+`yf.download()` and `Ticker.history()` have many options for configuring fetching and processing, e.g.:
 
 ```python
-data = yf.download(  # or pdr.get_data_yahoo(...
-        # tickers list or string as well
-        tickers = "SPY AAPL MSFT",
-
-        # use "period" instead of start/end
-        # valid periods: 1d,5d,1mo,3mo,6mo,1y,2y,5y,10y,ytd,max
-        # (optional, default is '1mo')
-        period = "ytd",
-
-        # fetch data by interval (including intraday if period < 60 days)
-        # valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
-        # (optional, default is '1d')
-        interval = "5d",
-
-        # Whether to ignore timezone when aligning ticker data from 
-        # different timezones. Default is False.
-        ignore_tz = False,
-
-        # group by ticker (to access via data['SPY'])
-        # (optional, default is 'column')
-        group_by = 'ticker',
-
-        # adjust all OHLC automatically
-        # (optional, default is False)
-        auto_adjust = True,
-
-        # attempt repair of Yahoo data issues
-        repair = False,
-
-        # download pre/post regular market hours data
-        # (optional, default is False)
-        prepost = True,
-
-        # use threads for mass downloading? (True/False/Integer)
-        # (optional, default is True)
-        threads = True,
-
-        # proxy URL scheme use use when downloading?
-        # (optional, default is None)
-        proxy = None
-    )
+yf.download(tickers = "SPY AAPL",  # list of tickers
+            period = "1y",         # time period
+            interval = "1d",       # trading interval
+            ignore_tz = True,      # ignore timezone when aligning data from different exchanges?
+            prepost = False)       # download pre/post market hours data?
 ```
 
-### Timezone cache store
+Review the [Wiki](https://github.com/ranaroussi/yfinance/wiki) for more options and detail.
+
+### Smarter scraping
+
+To use a custom `requests` session (for example to cache calls to the
+API or customize the `User-agent` header), pass a `session=` argument to
+the Ticker constructor.
 
-When fetching price data, all dates are localized to stock exchange timezone. 
-But timezone retrieval is relatively slow, so yfinance attemps to cache them 
-in your users cache folder. 
-You can direct cache to use a different location with `set_tz_cache_location()`:
 ```python
-import yfinance as yf
-yf.set_tz_cache_location("custom/cache/location")
-...
+import requests_cache
+session = requests_cache.CachedSession('yfinance.cache')
+session.headers['User-agent'] = 'my-program/1.0'
+ticker = yf.Ticker('msft', session=session)
+# The scraped response will be stored in the cache
+ticker.actions
+```
+
+Combine a `requests_cache` with rate-limiting to avoid triggering Yahoo's rate-limiter/blocker that can corrupt data.
+```python
+from requests import Session
+from requests_cache import CacheMixin, SQLiteCache
+from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
+class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
+    """ """
+
+session = CachedLimiterSession(
+    per_second=0.9,
+    bucket_class=MemoryQueueBucket,
+    backend=SQLiteCache("yfinance.cache"),
+)
 ```
 
 ### Managing Multi-Level Columns
@@ -260,6 +231,18 @@ yfinance?](https://stackoverflow.com/questions/63107801)
         -   How to download single or multiple tickers into a single
             dataframe with single level column names and a ticker column
 
+### Timezone cache store
+
+When fetching price data, all dates are localized to stock exchange timezone. 
+But timezone retrieval is relatively slow, so yfinance attemps to cache them 
+in your users cache folder. 
+You can direct cache to use a different location with `set_tz_cache_location()`:
+```python
+import yfinance as yf
+yf.set_tz_cache_location("custom/cache/location")
+...
+```
+
 ---
 
 ## `pandas_datareader` override

meta.yaml

@@ -1,5 +1,5 @@
 {% set name = "yfinance" %}
-{% set version = "0.2.9" %}
+{% set version = "0.2.12" %}
 
 package:
   name: "{{ name|lower }}"

tests/ticker.py

@@ -52,12 +52,16 @@ class TestTicker(unittest.TestCase):
     def test_badTicker(self):
         # Check yfinance doesn't die when ticker delisted
 
-        tkr = "AM2Z.TA"
+        tkr = "DJI"  # typo of "^DJI"
         dat = yf.Ticker(tkr, session=self.session)
         dat.history(period="1wk")
         dat.history(start="2022-01-01")
         dat.history(start="2022-01-01", end="2022-03-01")
         yf.download([tkr], period="1wk")
+
+        for k in dat.fast_info:
+            dat.fast_info[k]
+
         dat.isin
         dat.major_holders
         dat.institutional_holders
@@ -91,43 +95,48 @@ class TestTicker(unittest.TestCase):
     def test_goodTicker(self):
         # that yfinance works when full api is called on same instance of ticker
 
-        tkr = "IBM"
-        dat = yf.Ticker(tkr, session=self.session)
+        tkrs = ["IBM"]
+        tkrs.append("QCSTIX")  # weird ticker, no price history but has previous close
+        for tkr in tkrs:
+            dat = yf.Ticker(tkr, session=self.session)
 
-        dat.isin
-        dat.major_holders
-        dat.institutional_holders
-        dat.mutualfund_holders
-        dat.dividends
-        dat.splits
-        dat.actions
-        dat.shares
-        dat.get_shares_full()
-        dat.info
-        dat.calendar
-        dat.recommendations
-        dat.earnings
-        dat.quarterly_earnings
-        dat.income_stmt
-        dat.quarterly_income_stmt
-        dat.balance_sheet
-        dat.quarterly_balance_sheet
-        dat.cashflow
-        dat.quarterly_cashflow
-        dat.recommendations_summary
-        dat.analyst_price_target
-        dat.revenue_forecasts
-        dat.sustainability
-        dat.options
-        dat.news
-        dat.earnings_trend
-        dat.earnings_dates
-        dat.earnings_forecasts
+            dat.history(period="1wk")
+            dat.history(start="2022-01-01")
+            dat.history(start="2022-01-01", end="2022-03-01")
+            yf.download([tkr], period="1wk")
 
-        dat.history(period="1wk")
-        dat.history(start="2022-01-01")
-        dat.history(start="2022-01-01", end="2022-03-01")
-        yf.download([tkr], period="1wk")
+            for k in dat.fast_info:
+                dat.fast_info[k]
+
+            dat.isin
+            dat.major_holders
+            dat.institutional_holders
+            dat.mutualfund_holders
+            dat.dividends
+            dat.splits
+            dat.actions
+            dat.shares
+            dat.get_shares_full()
+            dat.info
+            dat.calendar
+            dat.recommendations
+            dat.earnings
+            dat.quarterly_earnings
+            dat.income_stmt
+            dat.quarterly_income_stmt
+            dat.balance_sheet
+            dat.quarterly_balance_sheet
+            dat.cashflow
+            dat.quarterly_cashflow
+            dat.recommendations_summary
+            dat.analyst_price_target
+            dat.revenue_forecasts
+            dat.sustainability
+            dat.options
+            dat.news
+            dat.earnings_trend
+            dat.earnings_dates
+            dat.earnings_forecasts
 
 
 class TestTickerHistory(unittest.TestCase):

yfinance/base.py

@@ -270,11 +270,15 @@ class FastInfo:
             return self._last_price
         prices = self._get_1y_prices()
         if prices.empty:
-            self._last_price = self._get_exchange_metadata()["regularMarketPrice"]
+            md = self._get_exchange_metadata()
+            if "regularMarketPrice" in md:
+                self._last_price = md["regularMarketPrice"]
         else:
             self._last_price = float(prices["Close"].iloc[-1])
             if _np.isnan(self._last_price):
-                self._last_price = self._get_exchange_metadata()["regularMarketPrice"]
+                md = self._get_exchange_metadata()
+                if "regularMarketPrice" in md:
+                    self._last_price = md["regularMarketPrice"]
         return self._last_price
 
     @property
@@ -282,16 +286,23 @@ class FastInfo:
         if self._prev_close is not None:
             return self._prev_close
         prices = self._get_1wk_1h_prepost_prices()
-        prices = prices[["Close"]].groupby(prices.index.date).last()
-        if prices.shape[0] < 2:
-            # Very few symbols have previousClose despite no 
-            # no trading data. E.g. 'QCSTIX'.
-            # So fallback to original info[] if available.
-            self._tkr.info  # trigger fetch
-            if "previousClose" in self._tkr._quote._retired_info:
-                self._prev_close = self._tkr._quote._retired_info["previousClose"]
+        fail = False
+        if prices.empty:
+            fail = True
         else:
-            self._prev_close = float(prices["Close"].iloc[-2])
+            prices = prices[["Close"]].groupby(prices.index.date).last()
+            if prices.shape[0] < 2:
+                # Very few symbols have previousClose despite no 
+                # no trading data e.g. 'QCSTIX'.
+                fail = True
+            else:
+                self._prev_close = float(prices["Close"].iloc[-2])
+        if fail:
+            # Fallback to original info[] if available.
+            self._tkr.info  # trigger fetch
+            k = "previousClose"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._prev_close = self._tkr._quote._retired_info[k]
         return self._prev_close
 
     @property
@@ -309,8 +320,9 @@ class FastInfo:
             # no trading data. E.g. 'QCSTIX'.
             # So fallback to original info[] if available.
             self._tkr.info  # trigger fetch
-            if "regularMarketPreviousClose" in self._tkr._quote._retired_info:
-                self._reg_prev_close = self._tkr._quote._retired_info["regularMarketPreviousClose"]
+            k = "regularMarketPreviousClose"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._reg_prev_close = self._tkr._quote._retired_info[k]
         else:
             self._reg_prev_close = float(prices["Close"].iloc[-2])
         return self._reg_prev_close
@@ -483,8 +495,9 @@ class FastInfo:
             # E.g. 'BTC-USD'
             # So fallback to original info[] if available.
             self._tkr.info
-            if "marketCap" in self._tkr._quote._retired_info:
-                self._mcap = self._tkr._quote._retired_info["marketCap"]
+            k = "marketCap"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._mcap = self._tkr._quote._retired_info[k]
         else:
             self._mcap = float(shares * self.last_price)
         return self._mcap

yfinance/data.py

@@ -250,15 +250,16 @@ class TickerData:
                 response_js.close()
             if len(re_keys) == key_count:
                 break
-        re_obj = {}
-        missing_val = False
-        for k in re_keys:
-            if not re_data.get(k):
-                missing_val = True
-                break
-            re_obj.update({k: re_data.get(k)})
-        if not missing_val:
-            return [''.join(re_obj.values())]
+        if len(re_keys) > 0:
+            re_obj = {}
+            missing_val = False
+            for k in re_keys:
+                if not re_data.get(k):
+                    missing_val = True
+                    break
+                re_obj.update({k: re_data.get(k)})
+            if not missing_val:
+                return [''.join(re_obj.values())]
 
         return []
 
@@ -289,11 +290,11 @@ class TickerData:
         # Gather decryption keys:
         soup = BeautifulSoup(response.content, "html.parser")
         keys = self._get_decryption_keys_from_yahoo_js(soup)
-        if len(keys) == 0:
-            msg = "No decryption keys could be extracted from JS file."
-            if "requests_cache" in str(type(response)):
-                msg += " Try flushing your 'requests_cache', probably parsing old JS."
-            print("WARNING: " + msg + " Falling back to backup decrypt methods.")
+        # if len(keys) == 0:
+        #     msg = "No decryption keys could be extracted from JS file."
+        #     if "requests_cache" in str(type(response)):
+        #         msg += " Try flushing your 'requests_cache', probably parsing old JS."
+        #     print("WARNING: " + msg + " Falling back to backup decrypt methods.")
         if len(keys) == 0:
             keys = []
             try:

yfinance/scrapers/yahoo-keys.txt

@@ -3,3 +3,6 @@ ad4d90b3c9f2e1d156ef98eadfa0ff93e4042f6960e54aa2a13f06f528e6b50ba4265a26a1fd5b9c
 e9a8ab8e5620b712ebc2fb4f33d5c8b9c80c0d07e8c371911c785cf674789f1747d76a909510158a7b7419e86857f2d7abbd777813ff64840e4cbc514d12bcae
 6ae2523aeafa283dad746556540145bf603f44edbf37ad404d3766a8420bb5eb1d3738f52a227b88283cca9cae44060d5f0bba84b6a495082589f5fe7acbdc9e
 3365117c2a368ffa5df7313a4a84988f73926a86358e8eea9497c5ff799ce27d104b68e5f2fbffa6f8f92c1fef41765a7066fa6bcf050810a9c4c7872fd3ebf0
+15d8f57919857d5a5358d2082c7ef0f1129cfacd2a6480333dcfb954b7bb67d820abefebfdb0eaa6ef18a1c57f617b67d7e7b0ec040403b889630ae5db5a4dbb
+db9630d707a7d0953ac795cd8db1ca9ca6c9d8239197cdfda24b4e0ec9c37eaec4db82dab68b8f606ab7b5b4af3e65dab50606f8cf508269ec927e6ee605fb78
+3c895fb5ddcc37d20d3073ed74ee3efad59bcb147c8e80fd279f83701b74b092d503dcd399604c6d8be8f3013429d3c2c76ed5b31b80c9df92d5eab6d3339fce

yfinance/utils.py

@@ -699,7 +699,7 @@ def format_history_metadata(md):
     tz = md["exchangeTimezoneName"]
 
     for k in ["firstTradeDate", "regularMarketTime"]:
-        if k in md:
+        if k in md and md[k] is not None:
             md[k] = _pd.to_datetime(md[k], unit='s', utc=True).tz_convert(tz)
 
     if "currentTradingPeriod" in md:

yfinance/version.py

@@ -1 +1 @@
-version = "0.2.10b3"
+version = "0.2.12"