Bump version to 0.2.19

dev -> main

.github/ISSUE_TEMPLATE/bug_report.md

@@ -9,7 +9,7 @@ assignees: ''
 
 # IMPORTANT
 
-If you want help, you got to read this first, follow the instructions.
+# Read and follow these instructions carefully. Help us help you.
 
 ### Are you up-to-date?
 
@@ -25,19 +25,20 @@ and comparing against [PIP](https://pypi.org/project/yfinance/#history).
 
 ### Does Yahoo actually have the data?
 
-Are you spelling ticker *exactly* same as Yahoo?
+Are you spelling symbol *exactly* same as Yahoo?
 
-Then visit `finance.yahoo.com` and confirm they have the data you want. Maybe your ticker was delisted, or your expectations of `yfinance` are wrong.
+Then visit `finance.yahoo.com` and confirm they have the data you want. Maybe your symbol was delisted, or your expectations of `yfinance` are wrong.
 
 ### Are you spamming Yahoo?
 
-Yahoo Finance free service has rate-limiting depending on request type - roughly 60/minute for prices, 10/minute for info. Once limit hit, Yahoo can delay, block, or return bad data. Not a `yfinance` bug.
+Yahoo Finance free service has rate-limiting depending on request type - roughly 60/minute for prices, 10/minute for info. Once limit hit, Yahoo can delay, block, or return bad data -> not a `yfinance` bug.
 
 ### Still think it's a bug?
 
-Delete this default message (all of it) and submit your bug report here, providing the following as best you can:
+**Delete these instructions** and replace with your bug report, providing the following as best you can:
 
-- Simple code that reproduces your problem, that we can copy-paste-run
-- Exception message with full traceback, or proof `yfinance` returning bad data
-- `yfinance` version and Python version
-- Operating system type
+- Simple code that reproduces your problem, that we can copy-paste-run.
+- Run code with [debug logging enabled](https://github.com/ranaroussi/yfinance/tree/dev#logging) and post the full output.
+- If you think `yfinance` returning bad data, give us proof.
+- `yfinance` version and Python version.
+- Operating system type.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,14 +0,0 @@
----
-name: Feature request
-about: Request a new feature
-title: ''
-labels: ''
-assignees: ''
-
----
-
-**Describe the problem**
-
-**Describe the solution**
-
-**Additional context**

CHANGELOG.rst

@@ -1,6 +1,41 @@
 Change Log
 ===========
 
+0.2.19
+------
+Switch to `logging` module #1493 #1522 #1541
+Price history:
+- optimise #1514
+- fixes #1523
+- fix TZ-cache corruption #1528
+
+0.2.18
+------
+Fix 'fast_info' error '_np not found' #1496
+Fix bug in timezone cache #1498
+
+0.2.17
+------
+Fix prices error with Pandas 2.0 #1488
+
+0.2.16
+------
+Fix 'fast_info deprecated' msg appearing at Ticker() init
+
+0.2.15
+------
+Restore missing Ticker.info keys #1480
+
+0.2.14
+------
+Fix Ticker.info dict by fetching from API #1461
+
+0.2.13
+------
+Price bug fixes:
+- fetch big-interval with Capital Gains #1455
+- merging dividends & splits with prices #1452
+
 0.2.12
 ------
 Disable annoying 'backup decrypt' msg

README.md

@@ -42,10 +42,19 @@ Yahoo! finance API is intended for personal use only.**
 
 ---
 
-## News [2023-01-27]
-Since December 2022 Yahoo has been encrypting the web data that `yfinance` scrapes for non-market data. Fortunately the decryption keys are available, although Yahoo moved/changed them several times hence `yfinance` breaking several times. `yfinance` is now better prepared for any future changes by Yahoo.
+## News
 
-Why is Yahoo doing this? We don't know. Is it to stop scrapers? Maybe, so we've implemented changes to reduce load on Yahoo. In December we rolled out version 0.2 with optimised scraping. Then in 0.2.6 introduced `Ticker.fast_info`, providing much faster access to some `info` elements wherever possible e.g. price stats and forcing users to switch (sorry but we think necessary). `info` will continue to exist for as long as there are elements without a fast alternative.
+### 2023-01-27
+Since December 2022 Yahoo has been encrypting the web data that `yfinance` scrapes for non-price data. Price data still works. Fortunately the decryption keys are available, although Yahoo moved/changed them several times hence `yfinance` breaking several times. `yfinance` is now better prepared for any future changes by Yahoo.
+
+Why is Yahoo doing this? We don't know. Is it to stop scrapers? Maybe, so we've implemented changes to reduce load on Yahoo. In December we rolled out version 0.2 with optimised scraping. Then in 0.2.6 introduced `Ticker.fast_info`, providing much faster access to some `Ticker.info` elements wherever possible e.g. price stats and forcing users to switch (sorry but we think necessary).
+
+### 2023-02-07
+Yahoo is now regularly changing their decryption key, breaking `yfinance` decryption. Is technically possible to extract this from their webpage but not implemented because difficult, see [discussion in the issue thread](https://github.com/ranaroussi/yfinance/issues/1407).
+
+### 2023-04-09
+
+Fixed `Ticker.info`
 
 ## Quick Start
 
@@ -58,10 +67,8 @@ import yfinance as yf
 
 msft = yf.Ticker("MSFT")
 
-# get all stock info (slow)
+# get all stock info
 msft.info
-# fast access to subset of stock info (opportunistic)
-msft.fast_info
 
 # get historical market data
 hist = msft.history(period="1mo")
@@ -154,6 +161,8 @@ msft.option_chain(..., proxy="PROXY_SERVER")
 ...
 ```
 
+### Multiple tickers
+
 To initialize multiple `Ticker` objects, use
 
 ```python
@@ -167,7 +176,7 @@ tickers.tickers['AAPL'].history(period="1mo")
 tickers.tickers['GOOG'].actions
 ```
 
-### Fetching data for multiple tickers
+To download price history into one table:
 
 ```python
 import yfinance as yf
@@ -180,12 +189,23 @@ data = yf.download("SPY AAPL", start="2017-01-01", end="2017-04-30")
 yf.download(tickers = "SPY AAPL",  # list of tickers
             period = "1y",         # time period
             interval = "1d",       # trading interval
-            ignore_tz = True,      # ignore timezone when aligning data from different exchanges?
-            prepost = False)       # download pre/post market hours data?
+            prepost = False,       # download pre/post market hours data?
+            repair = True)         # repair obvious price errors e.g. 100x?
 ```
 
 Review the [Wiki](https://github.com/ranaroussi/yfinance/wiki) for more options and detail.
 
+### Logging
+
+`yfinance` now uses the `logging` module. To control the detail of printed messages you simply change the level:
+```
+import logging
+logger = logging.getLogger('yfinance')
+logger.setLevel(logging.ERROR)  # default: only print errors
+logger.setLevel(logging.CRITICAL)  # disable printing
+logger.setLevel(logging.DEBUG)  # verbose: print errors & debug info
+```
+
 ### Smarter scraping
 
 To use a custom `requests` session (for example to cache calls to the
@@ -206,11 +226,12 @@ Combine a `requests_cache` with rate-limiting to avoid triggering Yahoo's rate-l
 from requests import Session
 from requests_cache import CacheMixin, SQLiteCache
 from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
+from pyrate_limiter import Duration, RequestRate, Limiter
 class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
-    """ """
+    pass
 
 session = CachedLimiterSession(
-    per_second=0.9,
+    limiter=Limiter(RequestRate(2, Duration.SECOND*5),  # max 2 requests per 5 seconds
     bucket_class=MemoryQueueBucket,
     backend=SQLiteCache("yfinance.cache"),
 )
@@ -231,21 +252,7 @@ yfinance?](https://stackoverflow.com/questions/63107801)
         -   How to download single or multiple tickers into a single
             dataframe with single level column names and a ticker column
 
-### Timezone cache store
-
-When fetching price data, all dates are localized to stock exchange timezone. 
-But timezone retrieval is relatively slow, so yfinance attemps to cache them 
-in your users cache folder. 
-You can direct cache to use a different location with `set_tz_cache_location()`:
-```python
-import yfinance as yf
-yf.set_tz_cache_location("custom/cache/location")
-...
-```
-
----
-
-## `pandas_datareader` override
+### `pandas_datareader` override
 
 If your code uses `pandas_datareader` and you want to download data
 faster, you can "hijack" `pandas_datareader.data.get_data_yahoo()`
@@ -262,6 +269,18 @@ yf.pdr_override() # <== that's all it takes :-)
 data = pdr.get_data_yahoo("SPY", start="2017-01-01", end="2017-04-30")
 ```
 
+### Timezone cache store
+
+When fetching price data, all dates are localized to stock exchange timezone. 
+But timezone retrieval is relatively slow, so yfinance attemps to cache them 
+in your users cache folder. 
+You can direct cache to use a different location with `set_tz_cache_location()`:
+```python
+import yfinance as yf
+yf.set_tz_cache_location("custom/cache/location")
+...
+```
+
 ---
 
 ## Installation
@@ -272,6 +291,11 @@ Install `yfinance` using `pip`:
 $ pip install yfinance --upgrade --no-cache-dir
 ```
 
+Test new features by installing betas, provide feedback in [corresponding Discussion](https://github.com/ranaroussi/yfinance/discussions):
+``` {.sourceCode .bash}
+$ pip install yfinance --upgrade --no-cache-dir --pre
+```
+
 To install `yfinance` using `conda`, see
 [this](https://anaconda.org/ranaroussi/yfinance).
 
@@ -289,11 +313,15 @@ To install `yfinance` using `conda`, see
 -   [html5lib](https://pypi.org/project/html5lib) \>= 1.1
 -   [cryptography](https://pypi.org/project/cryptography) \>= 3.3.2
 
-### Optional (if you want to use `pandas_datareader`)
+#### Optional (if you want to use `pandas_datareader`)
 
 -   [pandas\_datareader](https://github.com/pydata/pandas-datareader)
     \>= 0.4.0
 
+## Developers: want to contribute?
+
+`yfinance` relies on community to investigate bugs and contribute code. Developer guide: https://github.com/ranaroussi/yfinance/discussions/1084
+
 ---
 
 ### Legal Stuff

meta.yaml

@@ -1,5 +1,5 @@
 {% set name = "yfinance" %}
-{% set version = "0.2.12" %}
+{% set version = "0.2.19" %}
 
 package:
   name: "{{ name|lower }}"

test_yfinance.py

@@ -15,6 +15,9 @@ Sanity check for most common library uses all working
 
 import yfinance as yf
 import unittest
+import logging
+
+logging.basicConfig(level=logging.DEBUG)
 
 symbols = ['MSFT', 'IWO', 'VFINX', '^GSPC', 'BTC-USD']
 tickers = [yf.Ticker(symbol) for symbol in symbols]

tests/context.py

@@ -7,3 +7,32 @@ _src_dp = _parent_dp
 sys.path.insert(0, _src_dp)
 
 import yfinance
+
+
+# Optional: see the exact requests that are made during tests:
+# import logging
+# logging.basicConfig(level=logging.DEBUG)
+
+
+# Setup a session to rate-limit and cache persistently:
+import datetime as _dt
+import os
+import appdirs as _ad
+from requests import Session
+from requests_cache import CacheMixin, SQLiteCache
+from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
+class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
+    pass
+from pyrate_limiter import Duration, RequestRate, Limiter
+history_rate = RequestRate(1, Duration.SECOND*2)
+limiter = Limiter(history_rate)
+session_gbl = CachedLimiterSession(
+    limiter=limiter,
+    bucket_class=MemoryQueueBucket,
+    backend=SQLiteCache(os.path.join(_ad.user_cache_dir(), "py-yfinance", "unittests-cache"), 
+    					expire_after=_dt.timedelta(hours=1)),
+)
+# Use this instead if only want rate-limiting:
+# from requests_ratelimiter import LimiterSession
+# session_gbl = LimiterSession(limiter=limiter)
+

tests/prices.py

@@ -1,4 +1,5 @@
 from .context import yfinance as yf
+from .context import session_gbl
 
 import unittest
 
@@ -7,15 +8,11 @@ import pytz as _tz
 import numpy as _np
 import pandas as _pd
 
-import requests_cache
-
 
 class TestPriceHistory(unittest.TestCase):
-    session = None
-
     @classmethod
     def setUpClass(cls):
-        cls.session = requests_cache.CachedSession(backend='memory')
+        cls.session = session_gbl
 
     @classmethod
     def tearDownClass(cls):
@@ -34,11 +31,23 @@ class TestPriceHistory(unittest.TestCase):
                 f = df.index.time == _dt.time(0)
                 self.assertTrue(f.all())
 
+    def test_download(self):
+        tkrs = ["BHP.AX", "IMP.JO", "BP.L", "PNL.L", "INTC"]
+        intervals = ["1d", "1wk", "1mo"]
+        for interval in intervals:
+            df = yf.download(tkrs, period="5y", interval=interval)
+
+            f = df.index.time == _dt.time(0)
+            self.assertTrue(f.all())
+
+            df_tkrs = df.columns.levels[1]
+            self.assertEqual(sorted(tkrs), sorted(df_tkrs))
+
     def test_duplicatingHourly(self):
         tkrs = ["IMP.JO", "BHG.JO", "SSW.JO", "BP.L", "INTC"]
         for tkr in tkrs:
             dat = yf.Ticker(tkr, session=self.session)
-            tz = dat._get_ticker_tz(debug_mode=False, proxy=None, timeout=None)
+            tz = dat._get_ticker_tz(proxy=None, timeout=None)
 
             dt_utc = _tz.timezone("UTC").localize(_dt.datetime.utcnow())
             dt = dt_utc.astimezone(_tz.timezone(tz))
@@ -58,7 +67,7 @@ class TestPriceHistory(unittest.TestCase):
         test_run = False
         for tkr in tkrs:
             dat = yf.Ticker(tkr, session=self.session)
-            tz = dat._get_ticker_tz(debug_mode=False, proxy=None, timeout=None)
+            tz = dat._get_ticker_tz(proxy=None, timeout=None)
 
             dt_utc = _tz.timezone("UTC").localize(_dt.datetime.utcnow())
             dt = dt_utc.astimezone(_tz.timezone(tz))
@@ -84,7 +93,7 @@ class TestPriceHistory(unittest.TestCase):
         test_run = False
         for tkr in tkrs:
             dat = yf.Ticker(tkr, session=self.session)
-            tz = dat._get_ticker_tz(debug_mode=False, proxy=None, timeout=None)
+            tz = dat._get_ticker_tz(proxy=None, timeout=None)
 
             dt = _tz.timezone(tz).localize(_dt.datetime.now())
             if dt.date().weekday() not in [1, 2, 3, 4]:
@@ -230,6 +239,11 @@ class TestPriceHistory(unittest.TestCase):
                 print("{}-without-events missing these dates: {}".format(tkr, missing_from_df2))
                 raise
 
+    def test_monthlyWithEvents2(self):
+        # Simply check no exception from internal merge
+        tkr = "ABBV"
+        yf.Ticker("ABBV").history(period="max", interval="1mo")
+
     def test_tz_dst_ambiguous(self):
         # Reproduce issue #1100
         try:
@@ -381,12 +395,22 @@ class TestPriceHistory(unittest.TestCase):
         df = dat.history(start=start, interval="1wk")
         self.assertTrue((df.index.weekday == 0).all())
 
+    def test_aggregate_capital_gains(self):
+        # Setup
+        tkr = "FXAIX"
+        dat = yf.Ticker(tkr, session=self.session)
+        start = "2017-12-31"
+        end = "2019-12-31"
+        interval = "3mo"
+
+        df = dat.history(start=start, end=end, interval=interval)
+
 class TestPriceRepair(unittest.TestCase):
     session = None
 
     @classmethod
     def setUpClass(cls):
-        cls.session = requests_cache.CachedSession(backend='memory')
+        cls.session = session_gbl
 
     @classmethod
     def tearDownClass(cls):
@@ -464,6 +488,9 @@ class TestPriceRepair(unittest.TestCase):
         f_1 = ratio == 1
         self.assertTrue((f_100 | f_1).all())
 
+        self.assertTrue("Repaired?" in df_repaired.columns)
+        self.assertFalse(df_repaired["Repaired?"].isna().any())
+
     def test_repair_100x_weekly_preSplit(self):
         # PNL.L has a stock-split in 2022. Sometimes requesting data before 2022 is not split-adjusted.
 
@@ -521,6 +548,9 @@ class TestPriceRepair(unittest.TestCase):
         f_1 = ratio == 1
         self.assertTrue((f_100 | f_1).all())
 
+        self.assertTrue("Repaired?" in df_repaired.columns)
+        self.assertFalse(df_repaired["Repaired?"].isna().any())
+
     def test_repair_100x_daily(self):
         tkr = "PNL.L"
         dat = yf.Ticker(tkr, session=self.session)
@@ -563,6 +593,9 @@ class TestPriceRepair(unittest.TestCase):
         f_1 = ratio == 1
         self.assertTrue((f_100 | f_1).all())
 
+        self.assertTrue("Repaired?" in df_repaired.columns)
+        self.assertFalse(df_repaired["Repaired?"].isna().any())
+
     def test_repair_zeroes_daily(self):
         tkr = "BBIL.L"
         dat = yf.Ticker(tkr, session=self.session)
@@ -590,6 +623,9 @@ class TestPriceRepair(unittest.TestCase):
         for c in ["Open", "Low", "High", "Close"]:
             self.assertTrue(_np.isclose(repaired_df[c], correct_df[c], rtol=1e-8).all())
 
+        self.assertTrue("Repaired?" in repaired_df.columns)
+        self.assertFalse(repaired_df["Repaired?"].isna().any())
+
     def test_repair_zeroes_hourly(self):
         tkr = "INTC"
         dat = yf.Ticker(tkr, session=self.session)
@@ -621,13 +657,8 @@ class TestPriceRepair(unittest.TestCase):
                 print(repaired_df[c] - correct_df[c])
                 raise
 
+        self.assertTrue("Repaired?" in repaired_df.columns)
+        self.assertFalse(repaired_df["Repaired?"].isna().any())
+
 if __name__ == '__main__':
     unittest.main()
-
-# # Run tests sequentially:
-# import inspect
-# test_src = inspect.getsource(TestPriceHistory)
-# unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: (
-#     test_src.index(f"def {x}") - test_src.index(f"def {y}")
-# )
-# unittest.main(verbosity=2)

yfinance/data.py

@@ -1,6 +1,7 @@
 import functools
 from functools import lru_cache
 
+import logging
 import hashlib
 from base64 import b64decode
 usePycryptodome = False  # slightly faster
@@ -15,6 +16,8 @@ else:
 import requests as requests
 import re
 from bs4 import BeautifulSoup
+import random
+import time
 
 from frozendict import frozendict
 
@@ -23,8 +26,12 @@ try:
 except ImportError:
     import json as json
 
+from . import utils
+
 cache_maxsize = 64
 
+logger = utils.get_yf_logger()
+
 
 def lru_cache_freezeargs(func):
     """
@@ -202,6 +209,11 @@ class TickerData:
             proxy = {"https": proxy}
         return proxy
 
+    def get_raw_json(self, url, user_agent_headers=None, params=None, proxy=None, timeout=30):
+        response = self.get(url, user_agent_headers=user_agent_headers, params=params, proxy=proxy, timeout=timeout)
+        response.raise_for_status()
+        return response.json()
+
     def _get_decryption_keys_from_yahoo_js(self, soup):
         result = None
 
@@ -290,11 +302,11 @@ class TickerData:
         # Gather decryption keys:
         soup = BeautifulSoup(response.content, "html.parser")
         keys = self._get_decryption_keys_from_yahoo_js(soup)
-        # if len(keys) == 0:
-        #     msg = "No decryption keys could be extracted from JS file."
-        #     if "requests_cache" in str(type(response)):
-        #         msg += " Try flushing your 'requests_cache', probably parsing old JS."
-        #     print("WARNING: " + msg + " Falling back to backup decrypt methods.")
+        if len(keys) == 0:
+            msg = "No decryption keys could be extracted from JS file."
+            if "requests_cache" in str(type(response)):
+                msg += " Try flushing your 'requests_cache', probably parsing old JS."
+            logger.warning("%s Falling back to backup decrypt methods.", msg)
         if len(keys) == 0:
             keys = []
             try:

yfinance/multi.py

@@ -21,6 +21,8 @@
 
 from __future__ import print_function
 
+import logging
+import traceback
 import time as _time
 import multitasking as _multitasking
 import pandas as _pd
@@ -28,11 +30,10 @@ import pandas as _pd
 from . import Ticker, utils
 from . import shared
 
-
 def download(tickers, start=None, end=None, actions=False, threads=True, ignore_tz=None,
              group_by='column', auto_adjust=False, back_adjust=False, repair=False, keepna=False,
-             progress=True, period="max", show_errors=True, interval="1d", prepost=False,
-             proxy=None, rounding=False, timeout=10):
+             progress=True, period="max", show_errors=None, interval="1d", prepost=False,
+             proxy=None, rounding=False, timeout=10, session=None):
     """Download yahoo tickers
     :Parameters:
         tickers : str, list
@@ -44,11 +45,13 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
             Valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
             Intraday data cannot extend last 60 days
         start: str
-            Download start date string (YYYY-MM-DD) or _datetime.
+            Download start date string (YYYY-MM-DD) or _datetime, inclusive.
             Default is 1900-01-01
+            E.g. for start="2020-01-01", the first data point will be on "2020-01-01"
         end: str
-            Download end date string (YYYY-MM-DD) or _datetime.
+            Download end date string (YYYY-MM-DD) or _datetime, exclusive.
             Default is now
+            E.g. for end="2023-01-01", the last data point will be on "2022-12-31"
         group_by : str
             Group by 'ticker' or 'column' (default)
         prepost : bool
@@ -75,11 +78,22 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
             Optional. Round values to 2 decimal places?
         show_errors: bool
             Optional. Doesn't print errors if False
+            DEPRECATED, will be removed in future version
         timeout: None or float
             If not None stops waiting for a response after given number of
             seconds. (Can also be a fraction of a second e.g. 0.01)
+        session: None or Session
+            Optional. Pass your own session object to be used for all requests
     """
 
+    if show_errors is not None:
+        if show_errors:
+            utils.print_once(f"yfinance: download(show_errors={show_errors}) argument is deprecated and will be removed in future version. Do this instead: logging.getLogger('yfinance').setLevel(logging.ERROR)")
+            logging.getLogger('yfinance').setLevel(logging.ERROR)
+        else:
+            utils.print_once(f"yfinance: download(show_errors={show_errors}) argument is deprecated and will be removed in future version. Do this instead to suppress error messages: logging.getLogger('yfinance').setLevel(logging.CRITICAL)")
+            logging.getLogger('yfinance').setLevel(logging.CRITICAL)
+
     if ignore_tz is None:
         # Set default value depending on interval
         if interval[1:] in ['m', 'h']:
@@ -98,7 +112,7 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
     for ticker in tickers:
         if utils.is_isin(ticker):
             isin = ticker
-            ticker = utils.get_ticker_by_isin(ticker, proxy)
+            ticker = utils.get_ticker_by_isin(ticker, proxy, session=session)
             shared._ISINS[ticker] = isin
         _tickers_.append(ticker)
 
@@ -112,6 +126,7 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
     # reset shared._DFS
     shared._DFS = {}
     shared._ERRORS = {}
+    shared._TRACEBACKS = {}
 
     # download using threads
     if threads:
@@ -124,10 +139,9 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
                                    actions=actions, auto_adjust=auto_adjust,
                                    back_adjust=back_adjust, repair=repair, keepna=keepna,
                                    progress=(progress and i > 0), proxy=proxy,
-                                   rounding=rounding, timeout=timeout)
+                                   rounding=rounding, timeout=timeout, session=session)
         while len(shared._DFS) < len(tickers):
             _time.sleep(0.01)
-
     # download synchronously
     else:
         for i, ticker in enumerate(tickers):
@@ -136,20 +150,40 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
                                  actions=actions, auto_adjust=auto_adjust,
                                  back_adjust=back_adjust, repair=repair, keepna=keepna,
                                  proxy=proxy,
-                                 rounding=rounding, timeout=timeout)
-            shared._DFS[ticker.upper()] = data
+                                 rounding=rounding, timeout=timeout, session=session)
             if progress:
                 shared._PROGRESS_BAR.animate()
-
+    
     if progress:
         shared._PROGRESS_BAR.completed()
 
-    if shared._ERRORS and show_errors:
-        print('\n%.f Failed download%s:' % (
+    if shared._ERRORS:
+        # Send errors to logging module
+        logger = utils.get_yf_logger()
+        logger.error('\n%.f Failed download%s:' % (
             len(shared._ERRORS), 's' if len(shared._ERRORS) > 1 else ''))
-        # print(shared._ERRORS)
-        print("\n".join(['- %s: %s' %
-                         v for v in list(shared._ERRORS.items())]))
+
+        # Log each distinct error once, with list of symbols affected
+        errors = {}
+        for ticker in shared._ERRORS:
+            err = shared._ERRORS[ticker]
+            if not err in errors:
+                errors[err] = [ticker]
+            else:
+                errors[err].append(ticker)
+        for err in errors.keys():
+            logger.error(f'{errors[err]}: ' + err)
+
+        # Log each distinct traceback once, with list of symbols affected
+        tbs = {}
+        for ticker in shared._TRACEBACKS:
+            tb = shared._TRACEBACKS[ticker]
+            if not tb in tbs:
+                tbs[tb] = [ticker]
+            else:
+                tbs[tb].append(ticker)
+        for tb in tbs.keys():
+            logger.debug(f'{tbs[tb]}: ' + tb)
 
     if ignore_tz:
         for tkr in shared._DFS.keys():
@@ -206,17 +240,10 @@ def _download_one_threaded(ticker, start=None, end=None,
                            auto_adjust=False, back_adjust=False, repair=False,
                            actions=False, progress=True, period="max",
                            interval="1d", prepost=False, proxy=None,
-                           keepna=False, rounding=False, timeout=10):
-    try:
-        data = _download_one(ticker, start, end, auto_adjust, back_adjust, repair,
-                             actions, period, interval, prepost, proxy, rounding,
-                             keepna, timeout)
-    except Exception as e:
-        # glob try/except needed as current thead implementation breaks if exception is raised.
-        shared._DFS[ticker] = utils.empty_df()
-        shared._ERRORS[ticker] = repr(e)
-    else:
-        shared._DFS[ticker.upper()] = data
+                           keepna=False, rounding=False, timeout=10, session=None):
+    data = _download_one(ticker, start, end, auto_adjust, back_adjust, repair,
+                         actions, period, interval, prepost, proxy, rounding,
+                         keepna, timeout, session)
     if progress:
         shared._PROGRESS_BAR.animate()
 
@@ -225,12 +252,23 @@ def _download_one(ticker, start=None, end=None,
                   auto_adjust=False, back_adjust=False, repair=False,
                   actions=False, period="max", interval="1d",
                   prepost=False, proxy=None, rounding=False,
-                  keepna=False, timeout=10):
-    return Ticker(ticker).history(
-        period=period, interval=interval,
-        start=start, end=end, prepost=prepost,
-        actions=actions, auto_adjust=auto_adjust,
-        back_adjust=back_adjust, repair=repair, proxy=proxy,
-        rounding=rounding, keepna=keepna, timeout=timeout,
-        debug=False, raise_errors=False  # debug and raise_errors false to not log and raise errors in threads
-    )
+                  keepna=False, timeout=10, session=None):
+    data = None
+    try:
+        data = Ticker(ticker, session=session).history(
+                period=period, interval=interval,
+                start=start, end=end, prepost=prepost,
+                actions=actions, auto_adjust=auto_adjust,
+                back_adjust=back_adjust, repair=repair, proxy=proxy,
+                rounding=rounding, keepna=keepna, timeout=timeout,
+                raise_errors=True
+        )
+    except Exception as e:
+        # glob try/except needed as current thead implementation breaks if exception is raised.
+        shared._DFS[ticker.upper()] = utils.empty_df()
+        shared._ERRORS[ticker.upper()] = repr(e)
+        shared._TRACEBACKS[ticker.upper()] = traceback.format_exc()
+    else:
+        shared._DFS[ticker.upper()] = data
+
+    return data

yfinance/scrapers/analysis.py

@@ -58,7 +58,7 @@ class Analysis:
             analysis_data = analysis_data['QuoteSummaryStore']
         except KeyError as e:
             err_msg = "No analysis data found, symbol may be delisted"
-            print('- %s: %s' % (self._data.ticker, err_msg))
+            logger.error('%s: %s', self._data.ticker, err_msg)
             return
 
         if isinstance(analysis_data.get('earningsTrend'), dict):

yfinance/scrapers/fundamentals.py

@@ -1,4 +1,5 @@
 import datetime
+import logging
 import json
 
 import pandas as pd
@@ -8,6 +9,7 @@ from yfinance import utils
 from yfinance.data import TickerData
 from yfinance.exceptions import YFinanceDataException, YFinanceException
 
+logger = utils.get_yf_logger()
 
 class Fundamentals:
 
@@ -50,7 +52,7 @@ class Fundamentals:
             self._fin_data_quote = self._financials_data['QuoteSummaryStore']
         except KeyError:
             err_msg = "No financials data found, symbol may be delisted"
-            print('- %s: %s' % (self._data.ticker, err_msg))
+            logger.error('%s: %s', self._data.ticker, err_msg)
             return None
 
     def _scrape_earnings(self, proxy):
@@ -144,7 +146,7 @@ class Financials:
             if statement is not None:
                 return statement
         except YFinanceException as e:
-            print(f"- {self._data.ticker}: Failed to create {name} financials table for reason: {repr(e)}")
+            logger.error("%s: Failed to create %s financials table for reason: %r", self._data.ticker, name, e)
         return pd.DataFrame()
 
     def _create_financials_table(self, name, timescale, proxy):
@@ -267,7 +269,7 @@ class Financials:
             if statement is not None:
                 return statement
         except YFinanceException as e:
-            print(f"- {self._data.ticker}: Failed to create financials table for {name} reason: {repr(e)}")
+            logger.error("%s: Failed to create financials table for %s reason: %r", self._data.ticker, name, e)
         return pd.DataFrame()
 
     def _create_financials_table_old(self, name, timescale, proxy):

yfinance/scrapers/quote.py

@@ -1,11 +1,15 @@
 import datetime
+import logging
 import json
+import warnings
 
 import pandas as pd
+import numpy as _np
 
 from yfinance import utils
 from yfinance.data import TickerData
 
+logger = utils.get_yf_logger()
 
 info_retired_keys_price = {"currentPrice", "dayHigh", "dayLow", "open", "previousClose", "volume", "volume24Hr"}
 info_retired_keys_price.update({"regularMarket"+s for s in ["DayHigh", "DayLow", "Open", "PreviousClose", "Price", "Volume"]})
@@ -19,6 +23,7 @@ info_retired_keys = info_retired_keys_price | info_retired_keys_exchange | info_
 
 PRUNE_INFO = True
 # PRUNE_INFO = False
+_BASIC_URL_ = "https://query2.finance.yahoo.com/v10/finance/quoteSummary"
 
 
 from collections.abc import MutableMapping
@@ -44,16 +49,16 @@ class InfoDictWrapper(MutableMapping):
 
     def __getitem__(self, k):
         if k in info_retired_keys_price:
-            print(f"Price data removed from info (key='{k}'). Use Ticker.fast_info or history() instead")
+            warnings.warn(f"Price data removed from info (key='{k}'). Use Ticker.fast_info or history() instead", DeprecationWarning)
             return None
         elif k in info_retired_keys_exchange:
-            print(f"Exchange data removed from info (key='{k}'). Use Ticker.fast_info or Ticker.get_history_metadata() instead")
+            warnings.warn(f"Exchange data removed from info (key='{k}'). Use Ticker.fast_info or Ticker.get_history_metadata() instead", DeprecationWarning)
             return None
         elif k in info_retired_keys_marketCap:
-            print(f"Market cap removed from info (key='{k}'). Use Ticker.fast_info instead")
+            warnings.warn(f"Market cap removed from info (key='{k}'). Use Ticker.fast_info instead", DeprecationWarning)
             return None
         elif k in info_retired_keys_symbol:
-            print(f"Symbol removed from info (key='{k}'). You know this already")
+            warnings.warn(f"Symbol removed from info (key='{k}'). You know this already", DeprecationWarning)
             return None
         return self.info[self._keytransform(k)]
 
@@ -73,6 +78,476 @@ class InfoDictWrapper(MutableMapping):
         return k
 
 
+class FastInfo:
+    # Contain small subset of info[] items that can be fetched faster elsewhere.
+    # Imitates a dict.
+    def __init__(self, tickerBaseObject):
+        utils.print_once("yfinance: Note: 'Ticker.info' dict is now fixed & improved, 'fast_info' is no longer faster")
+
+        self._tkr = tickerBaseObject
+
+        self._prices_1y = None
+        self._prices_1wk_1h_prepost = None
+        self._prices_1wk_1h_reg = None
+        self._md = None
+
+        self._currency = None
+        self._quote_type = None
+        self._exchange = None
+        self._timezone = None
+
+        self._shares = None
+        self._mcap = None
+
+        self._open = None
+        self._day_high = None
+        self._day_low = None
+        self._last_price = None
+        self._last_volume = None
+
+        self._prev_close = None
+
+        self._reg_prev_close = None
+
+        self._50d_day_average = None
+        self._200d_day_average = None
+        self._year_high = None
+        self._year_low = None
+        self._year_change = None
+
+        self._10d_avg_vol = None
+        self._3mo_avg_vol = None
+
+        # attrs = utils.attributes(self)
+        # self.keys = attrs.keys()
+        # utils.attributes is calling each method, bad! Have to hardcode
+        _properties = ["currency", "quote_type", "exchange", "timezone"]
+        _properties += ["shares", "market_cap"]
+        _properties += ["last_price", "previous_close", "open", "day_high", "day_low"]
+        _properties += ["regular_market_previous_close"]
+        _properties += ["last_volume"]
+        _properties += ["fifty_day_average", "two_hundred_day_average", "ten_day_average_volume", "three_month_average_volume"]
+        _properties += ["year_high", "year_low", "year_change"]
+
+        # Because released before fixing key case, need to officially support 
+        # camel-case but also secretly support snake-case
+        base_keys = [k for k in _properties if not '_' in k]
+
+        sc_keys = [k for k in _properties if '_' in k]
+
+        self._sc_to_cc_key = {k:utils.snake_case_2_camelCase(k) for k in sc_keys}
+        self._cc_to_sc_key = {v:k for k,v in self._sc_to_cc_key.items()}
+ 
+        self._public_keys = sorted(base_keys + list(self._sc_to_cc_key.values()))
+        self._keys = sorted(self._public_keys + sc_keys)
+
+    # dict imitation:
+    def keys(self):
+        return self._public_keys
+    def items(self):
+        return [(k,self[k]) for k in self._public_keys]
+    def values(self):
+        return [self[k] for k in self._public_keys]
+    def get(self, key, default=None):
+        if key in self.keys():
+            if key in self._cc_to_sc_key:
+                key = self._cc_to_sc_key[key]
+            return self[key]
+        return default
+    def __getitem__(self, k):
+        if not isinstance(k, str):
+            raise KeyError(f"key must be a string")
+        if not k in self._keys:
+            raise KeyError(f"'{k}' not valid key. Examine 'FastInfo.keys()'")
+        if k in self._cc_to_sc_key:
+            k = self._cc_to_sc_key[k]
+        return getattr(self, k)
+    def __contains__(self, k):
+        return k in self.keys()
+    def __iter__(self):
+        return iter(self.keys())
+
+    def __str__(self):
+        return "lazy-loading dict with keys = " + str(self.keys())
+    def __repr__(self):
+        return self.__str__()
+
+    def toJSON(self, indent=4):
+        d = {k:self[k] for k in self.keys()}
+        return _json.dumps({k:self[k] for k in self.keys()}, indent=indent)
+
+    def _get_1y_prices(self, fullDaysOnly=False):
+        if self._prices_1y is None:
+            # Temporarily disable error printing
+            l = logger.level
+            logger.setLevel(logging.CRITICAL)
+            self._prices_1y = self._tkr.history(period="380d", auto_adjust=False, keepna=True)
+            logger.setLevel(l)
+            self._md = self._tkr.get_history_metadata()
+            try:
+                ctp = self._md["currentTradingPeriod"]
+                self._today_open = pd.to_datetime(ctp["regular"]["start"], unit='s', utc=True).tz_convert(self.timezone)
+                self._today_close = pd.to_datetime(ctp["regular"]["end"], unit='s', utc=True).tz_convert(self.timezone)
+                self._today_midnight = self._today_close.ceil("D")
+            except:
+                self._today_open = None
+                self._today_close = None
+                self._today_midnight = None
+                raise
+
+        if self._prices_1y.empty:
+            return self._prices_1y
+
+        dnow = pd.Timestamp.utcnow().tz_convert(self.timezone).date()
+        d1 = dnow
+        d0 = (d1 + datetime.timedelta(days=1)) - utils._interval_to_timedelta("1y")
+        if fullDaysOnly and self._exchange_open_now():
+            # Exclude today
+            d1 -= utils._interval_to_timedelta("1d")
+        return self._prices_1y.loc[str(d0):str(d1)]
+
+    def _get_1wk_1h_prepost_prices(self):
+        if self._prices_1wk_1h_prepost is None:
+            # Temporarily disable error printing
+            l = logger.level
+            logger.setLevel(logging.CRITICAL)
+            self._prices_1wk_1h_prepost = self._tkr.history(period="1wk", interval="1h", auto_adjust=False, prepost=True)
+            logger.setLevel(l)
+        return self._prices_1wk_1h_prepost
+
+    def _get_1wk_1h_reg_prices(self):
+        if self._prices_1wk_1h_reg is None:
+            # Temporarily disable error printing
+            l = logger.level
+            logger.setLevel(logging.CRITICAL)
+            self._prices_1wk_1h_reg = self._tkr.history(period="1wk", interval="1h", auto_adjust=False, prepost=False)
+            logger.setLevel(l)
+        return self._prices_1wk_1h_reg
+
+    def _get_exchange_metadata(self):
+        if self._md is not None:
+            return self._md
+
+        self._get_1y_prices()
+        self._md = self._tkr.get_history_metadata()
+        return self._md
+
+    def _exchange_open_now(self):
+        t = pd.Timestamp.utcnow()
+        self._get_exchange_metadata()
+
+        # if self._today_open is None and self._today_close is None:
+        #     r = False
+        # else:
+        #     r = self._today_open <= t and t < self._today_close
+
+        # if self._today_midnight is None:
+        #     r = False
+        # elif self._today_midnight.date() > t.tz_convert(self.timezone).date():
+        #     r = False
+        # else:
+        #     r = t < self._today_midnight
+
+        last_day_cutoff = self._get_1y_prices().index[-1] + datetime.timedelta(days=1)
+        last_day_cutoff += datetime.timedelta(minutes=20)
+        r = t < last_day_cutoff
+
+        # print("_exchange_open_now() returning", r)
+        return r
+
+    @property
+    def currency(self):
+        if self._currency is not None:
+            return self._currency
+
+        if self._tkr._history_metadata is None:
+            self._get_1y_prices()
+        md = self._tkr.get_history_metadata()
+        self._currency = md["currency"]
+        return self._currency
+
+    @property
+    def quote_type(self):
+        if self._quote_type is not None:
+            return self._quote_type
+
+        if self._tkr._history_metadata is None:
+            self._get_1y_prices()
+        md = self._tkr.get_history_metadata()
+        self._quote_type = md["instrumentType"]
+        return self._quote_type
+
+    @property
+    def exchange(self):
+        if self._exchange is not None:
+            return self._exchange
+
+        self._exchange = self._get_exchange_metadata()["exchangeName"]
+        return self._exchange
+
+    @property
+    def timezone(self):
+        if self._timezone is not None:
+            return self._timezone
+
+        self._timezone = self._get_exchange_metadata()["exchangeTimezoneName"]
+        return self._timezone
+
+    @property
+    def shares(self):
+        if self._shares is not None:
+            return self._shares
+
+        shares = self._tkr.get_shares_full(start=pd.Timestamp.utcnow().date()-pd.Timedelta(days=548))
+        if shares is None:
+            # Requesting 18 months failed, so fallback to shares which should include last year
+            shares = self._tkr.get_shares()
+        if shares is not None:
+            if isinstance(shares, pd.DataFrame):
+                shares = shares[shares.columns[0]]
+            self._shares = int(shares.iloc[-1])
+        return self._shares
+
+    @property
+    def last_price(self):
+        if self._last_price is not None:
+            return self._last_price
+        prices = self._get_1y_prices()
+        if prices.empty:
+            md = self._get_exchange_metadata()
+            if "regularMarketPrice" in md:
+                self._last_price = md["regularMarketPrice"]
+        else:
+            self._last_price = float(prices["Close"].iloc[-1])
+            if _np.isnan(self._last_price):
+                md = self._get_exchange_metadata()
+                if "regularMarketPrice" in md:
+                    self._last_price = md["regularMarketPrice"]
+        return self._last_price
+
+    @property
+    def previous_close(self):
+        if self._prev_close is not None:
+            return self._prev_close
+        prices = self._get_1wk_1h_prepost_prices()
+        fail = False
+        if prices.empty:
+            fail = True
+        else:
+            prices = prices[["Close"]].groupby(prices.index.date).last()
+            if prices.shape[0] < 2:
+                # Very few symbols have previousClose despite no 
+                # no trading data e.g. 'QCSTIX'.
+                fail = True
+            else:
+                self._prev_close = float(prices["Close"].iloc[-2])
+        if fail:
+            # Fallback to original info[] if available.
+            self._tkr.info  # trigger fetch
+            k = "previousClose"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._prev_close = self._tkr._quote._retired_info[k]
+        return self._prev_close
+
+    @property
+    def regular_market_previous_close(self):
+        if self._reg_prev_close is not None:
+            return self._reg_prev_close
+        prices = self._get_1y_prices()
+        if prices.shape[0] == 1:
+            # Tiny % of tickers don't return daily history before last trading day, 
+            # so backup option is hourly history:
+            prices = self._get_1wk_1h_reg_prices()
+            prices = prices[["Close"]].groupby(prices.index.date).last()
+        if prices.shape[0] < 2:
+            # Very few symbols have regularMarketPreviousClose despite no 
+            # no trading data. E.g. 'QCSTIX'.
+            # So fallback to original info[] if available.
+            self._tkr.info  # trigger fetch
+            k = "regularMarketPreviousClose"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._reg_prev_close = self._tkr._quote._retired_info[k]
+        else:
+            self._reg_prev_close = float(prices["Close"].iloc[-2])
+        return self._reg_prev_close
+
+    @property
+    def open(self):
+        if self._open is not None:
+            return self._open
+        prices = self._get_1y_prices()
+        if prices.empty:
+            self._open = None
+        else:
+            self._open = float(prices["Open"].iloc[-1])
+            if _np.isnan(self._open):
+                self._open = None
+        return self._open
+
+    @property
+    def day_high(self):
+        if self._day_high is not None:
+            return self._day_high
+        prices = self._get_1y_prices()
+        if prices.empty:
+            self._day_high = None
+        else:
+            self._day_high = float(prices["High"].iloc[-1])
+            if _np.isnan(self._day_high):
+                self._day_high = None
+        return self._day_high
+
+    @property
+    def day_low(self):
+        if self._day_low is not None:
+            return self._day_low
+        prices = self._get_1y_prices()
+        if prices.empty:
+            self._day_low = None
+        else:
+            self._day_low = float(prices["Low"].iloc[-1])
+            if _np.isnan(self._day_low):
+                self._day_low = None
+        return self._day_low
+
+    @property
+    def last_volume(self):
+        if self._last_volume is not None:
+            return self._last_volume
+        prices = self._get_1y_prices()
+        self._last_volume = None if prices.empty else int(prices["Volume"].iloc[-1])
+        return self._last_volume
+
+    @property
+    def fifty_day_average(self):
+        if self._50d_day_average is not None:
+            return self._50d_day_average
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._50d_day_average = None
+        else:
+            n = prices.shape[0]
+            a = n-50
+            b = n
+            if a < 0:
+                a = 0
+            self._50d_day_average = float(prices["Close"].iloc[a:b].mean())
+
+        return self._50d_day_average
+
+    @property
+    def two_hundred_day_average(self):
+        if self._200d_day_average is not None:
+            return self._200d_day_average
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._200d_day_average = None
+        else:
+            n = prices.shape[0]
+            a = n-200
+            b = n
+            if a < 0:
+                a = 0
+
+            self._200d_day_average = float(prices["Close"].iloc[a:b].mean())
+
+        return self._200d_day_average
+
+    @property
+    def ten_day_average_volume(self):
+        if self._10d_avg_vol is not None:
+            return self._10d_avg_vol
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._10d_avg_vol = None
+        else:
+            n = prices.shape[0]
+            a = n-10
+            b = n
+            if a < 0:
+                a = 0
+            self._10d_avg_vol = int(prices["Volume"].iloc[a:b].mean())
+
+        return self._10d_avg_vol
+
+    @property
+    def three_month_average_volume(self):
+        if self._3mo_avg_vol is not None:
+            return self._3mo_avg_vol
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._3mo_avg_vol = None
+        else:
+            dt1 = prices.index[-1]
+            dt0 = dt1 - utils._interval_to_timedelta("3mo") + utils._interval_to_timedelta("1d")
+            self._3mo_avg_vol = int(prices.loc[dt0:dt1, "Volume"].mean())
+
+        return self._3mo_avg_vol
+
+    @property
+    def year_high(self):
+        if self._year_high is not None:
+            return self._year_high
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            prices = self._get_1y_prices(fullDaysOnly=False)
+        self._year_high = float(prices["High"].max())
+        return self._year_high
+
+    @property
+    def year_low(self):
+        if self._year_low is not None:
+            return self._year_low
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            prices = self._get_1y_prices(fullDaysOnly=False)
+        self._year_low = float(prices["Low"].min())
+        return self._year_low
+
+    @property
+    def year_change(self):
+        if self._year_change is not None:
+            return self._year_change
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.shape[0] >= 2:
+            self._year_change = (prices["Close"].iloc[-1] - prices["Close"].iloc[0]) / prices["Close"].iloc[0]
+            self._year_change = float(self._year_change)
+        return self._year_change
+
+    @property
+    def market_cap(self):
+        if self._mcap is not None:
+            return self._mcap
+
+        try:
+            shares = self.shares
+        except Exception as e:
+            if "Cannot retrieve share count" in str(e):
+                shares = None
+            elif "failed to decrypt Yahoo" in str(e):
+                shares = None
+            else:
+                raise
+
+        if shares is None:
+            # Very few symbols have marketCap despite no share count.
+            # E.g. 'BTC-USD'
+            # So fallback to original info[] if available.
+            self._tkr.info
+            k = "marketCap"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._mcap = self._tkr._quote._retired_info[k]
+        else:
+            self._mcap = float(shares * self.last_price)
+        return self._mcap
+
 
 class Quote:
 
@@ -87,13 +562,16 @@ class Quote:
         self._calendar = None
 
         self._already_scraped = False
-        self._already_scraped_complementary = False
+        self._already_fetched = False
+        self._already_fetched_complementary = False
 
     @property
     def info(self) -> dict:
         if self._info is None:
-            self._scrape(self.proxy)
-            self._scrape_complementary(self.proxy)
+            # self._scrape(self.proxy)  # decrypt broken
+            self._fetch(self.proxy)
+
+            self._fetch_complementary(self.proxy)
 
         return self._info
 
@@ -126,7 +604,7 @@ class Quote:
             quote_summary_store = json_data['QuoteSummaryStore']
         except KeyError:
             err_msg = "No summary info found, symbol may be delisted"
-            print('- %s: %s' % (self._data.ticker, err_msg))
+            logger.error('%s: %s', self._data.ticker, err_msg)
             return None
 
         # sustainability
@@ -236,12 +714,56 @@ class Quote:
         except Exception:
             pass
 
-    def _scrape_complementary(self, proxy):
-        if self._already_scraped_complementary:
+    def _fetch(self, proxy):
+        if self._already_fetched:
             return
-        self._already_scraped_complementary = True
+        self._already_fetched = True
+        modules = ['summaryProfile', 'financialData', 'quoteType',
+                     'defaultKeyStatistics', 'assetProfile', 'summaryDetail']
+        result = self._data.get_raw_json(
+            _BASIC_URL_ + f"/{self._data.ticker}", params={"modules": ",".join(modules), "ssl": "true"}, proxy=proxy
+        )
+        result["quoteSummary"]["result"][0]["symbol"] = self._data.ticker
+        query1_info = next(
+            (info for info in result.get("quoteSummary", {}).get("result", []) if info["symbol"] == self._data.ticker),
+            None,
+        )
+        # Most keys that appear in multiple dicts have same value. Except 'maxAge' because
+        # Yahoo not consistent with days vs seconds. Fix it here:
+        for k in query1_info:
+            if "maxAge" in query1_info[k] and query1_info[k]["maxAge"] == 1:
+                query1_info[k]["maxAge"] = 86400
+        query1_info = {
+            k1: v1 
+            for k, v in query1_info.items() 
+            if isinstance(v, dict) 
+            for k1, v1 in v.items() 
+            if v1
+        }
+        # recursively format but only because of 'companyOfficers'
+        def _format(k, v):
+            if isinstance(v, dict) and "raw" in v and "fmt" in v:
+                v2 = v["fmt"] if k in {"regularMarketTime", "postMarketTime"} else v["raw"]
+            elif isinstance(v, list):
+                v2 = [_format(None, x) for x in v]
+            elif isinstance(v, dict):
+                v2 = {k:_format(k, x) for k, x in v.items()}
+            elif isinstance(v, str):
+                v2 = v.replace("\xa0", " ")
+            else:
+                v2 = v
+            return v2
+        for k, v in query1_info.items():
+            query1_info[k] = _format(k, v)
+        self._info = query1_info
 
-        self._scrape(proxy)
+    def _fetch_complementary(self, proxy):
+        if self._already_fetched_complementary:
+            return
+        self._already_fetched_complementary = True
+
+        # self._scrape(proxy)  # decrypt broken
+        self._fetch(proxy)
         if self._info is None:
             return
 

yfinance/shared.py

@@ -22,4 +22,5 @@
 _DFS = {}
 _PROGRESS_BAR = None
 _ERRORS = {}
+_TRACEBACKS = {}
 _ISINS = {}

yfinance/tickers.py

@@ -87,10 +87,4 @@ class Tickers:
         return data
 
     def news(self):
-        collection = {}
-        for ticker in self.symbols:
-            collection[ticker] = []
-            items = Ticker(ticker).news
-            for item in items:
-                collection[ticker].append(item)
-        return collection
+        return {ticker: [item for item in Ticker(ticker).news] for ticker in self.symbols}

yfinance/utils.py

@@ -35,6 +35,8 @@ import os as _os
 import appdirs as _ad
 import sqlite3 as _sqlite3
 import atexit as _atexit
+from functools import lru_cache
+import logging
 
 from threading import Lock
 
@@ -61,6 +63,27 @@ def attributes(obj):
         if name[0] != '_' and name not in disallowed_names and hasattr(obj, name)}
 
 
+@lru_cache(maxsize=20)
+def print_once(msg):
+    # 'warnings' module suppression of repeat messages does not work. 
+    # This function replicates correct behaviour
+    print(msg)
+
+
+yf_logger = None
+def get_yf_logger():
+    global yf_logger
+    if yf_logger is None:
+        yf_logger = logging.getLogger("yfinance")
+        if yf_logger.handlers is None or len(yf_logger.handlers) == 0:
+            # Add stream handler if user not already added one
+            h = logging.StreamHandler()
+            formatter = logging.Formatter(fmt='%(levelname)s %(message)s')
+            h.setFormatter(formatter)
+            yf_logger.addHandler(h)
+    return yf_logger
+
+
 def is_isin(string):
     return bool(_re.match("^([A-Z]{2})([A-Z0-9]{9})([0-9]{1})$", string))
 
@@ -338,10 +361,10 @@ def _interval_to_timedelta(interval):
 def auto_adjust(data):
     col_order = data.columns
     df = data.copy()
-    ratio = df["Close"] / df["Adj Close"]
-    df["Adj Open"] = df["Open"] / ratio
-    df["Adj High"] = df["High"] / ratio
-    df["Adj Low"] = df["Low"] / ratio
+    ratio = (df["Adj Close"] / df["Close"]).to_numpy()
+    df["Adj Open"] = df["Open"] * ratio
+    df["Adj High"] = df["High"] * ratio
+    df["Adj Low"] = df["Low"] * ratio
 
     df.drop(
         ["Open", "High", "Low", "Close"],
@@ -404,12 +427,9 @@ def parse_quotes(data):
 
 
 def parse_actions(data):
-    dividends = _pd.DataFrame(
-        columns=["Dividends"], index=_pd.DatetimeIndex([]))
-    capital_gains = _pd.DataFrame(
-        columns=["Capital Gains"], index=_pd.DatetimeIndex([]))
-    splits = _pd.DataFrame(
-        columns=["Stock Splits"], index=_pd.DatetimeIndex([]))
+    dividends = None
+    capital_gains = None
+    splits = None
 
     if "events" in data:
         if "dividends" in data["events"]:
@@ -438,6 +458,16 @@ def parse_actions(data):
                                      splits["denominator"]
             splits = splits[["Stock Splits"]]
 
+    if dividends is None:
+        dividends = _pd.DataFrame(
+            columns=["Dividends"], index=_pd.DatetimeIndex([]))
+    if capital_gains is None:
+        capital_gains = _pd.DataFrame(
+            columns=["Capital Gains"], index=_pd.DatetimeIndex([]))
+    if splits is None:
+        splits = _pd.DataFrame(
+            columns=["Stock Splits"], index=_pd.DatetimeIndex([]))
+
     return dividends, splits, capital_gains
 
 
@@ -448,31 +478,30 @@ def set_df_tz(df, interval, tz):
     return df
 
 
-def fix_Yahoo_returning_prepost_unrequested(quotes, interval, metadata):
+def fix_Yahoo_returning_prepost_unrequested(quotes, interval, tradingPeriods):
     # Sometimes Yahoo returns post-market data despite not requesting it.
     # Normally happens on half-day early closes.
     #
     # And sometimes returns pre-market data despite not requesting it.
     # E.g. some London tickers.
-    tps_df = metadata["tradingPeriods"]
+    tps_df = tradingPeriods.copy()
     tps_df["_date"] = tps_df.index.date
     quotes["_date"] = quotes.index.date
     idx = quotes.index.copy()
-    quotes = quotes.merge(tps_df, how="left", validate="many_to_one")
+    quotes = quotes.merge(tps_df, how="left")
     quotes.index = idx
     # "end" = end of regular trading hours (including any auction)
     f_drop = quotes.index >= quotes["end"]
     f_drop = f_drop | (quotes.index < quotes["start"])
     if f_drop.any():
         # When printing report, ignore rows that were already NaNs:
-        f_na = quotes[["Open","Close"]].isna().all(axis=1)
-        n_nna = quotes.shape[0] - _np.sum(f_na)
-        n_drop_nna = _np.sum(f_drop & ~f_na)
-        quotes_dropped = quotes[f_drop]
+        # f_na = quotes[["Open","Close"]].isna().all(axis=1)
+        # n_nna = quotes.shape[0] - _np.sum(f_na)
+        # n_drop_nna = _np.sum(f_drop & ~f_na)
+        # quotes_dropped = quotes[f_drop]
         # if debug and n_drop_nna > 0:
         #     print(f"Dropping {n_drop_nna}/{n_nna} intervals for falling outside regular trading hours")
         quotes = quotes[~f_drop]
-    metadata["tradingPeriods"] = tps_df.drop(["_date"], axis=1)
     quotes = quotes.drop(["_date", "start", "end"], axis=1)
     return quotes
 
@@ -511,16 +540,24 @@ def fix_Yahoo_returning_live_separate(quotes, interval, tz_exchange):
                 # Last two rows are within same interval
                 idx1 = quotes.index[n - 1]
                 idx2 = quotes.index[n - 2]
+                if idx1 == idx2:
+                    # Yahoo returning last interval duplicated, which means
+                    # Yahoo is not returning live data (phew!)
+                    return quotes
                 if _np.isnan(quotes.loc[idx2, "Open"]):
                     quotes.loc[idx2, "Open"] = quotes["Open"][n - 1]
-                # Note: nanmax() & nanmin() ignores NaNs
-                quotes.loc[idx2, "High"] = _np.nanmax([quotes["High"][n - 1], quotes["High"][n - 2]])
-                quotes.loc[idx2, "Low"] = _np.nanmin([quotes["Low"][n - 1], quotes["Low"][n - 2]])
+                # Note: nanmax() & nanmin() ignores NaNs, but still need to check not all are NaN to avoid warnings
+                if not _np.isnan(quotes["High"][n - 1]):
+                    quotes.loc[idx2, "High"] = _np.nanmax([quotes["High"][n - 1], quotes["High"][n - 2]])
+                    if "Adj High" in quotes.columns:
+                        quotes.loc[idx2, "Adj High"] = _np.nanmax([quotes["Adj High"][n - 1], quotes["Adj High"][n - 2]])
+
+                if not _np.isnan(quotes["Low"][n - 1]):
+                    quotes.loc[idx2, "Low"] = _np.nanmin([quotes["Low"][n - 1], quotes["Low"][n - 2]])
+                    if "Adj Low" in quotes.columns:
+                        quotes.loc[idx2, "Adj Low"] = _np.nanmin([quotes["Adj Low"][n - 1], quotes["Adj Low"][n - 2]])
+
                 quotes.loc[idx2, "Close"] = quotes["Close"][n - 1]
-                if "Adj High" in quotes.columns:
-                    quotes.loc[idx2, "Adj High"] = _np.nanmax([quotes["Adj High"][n - 1], quotes["Adj High"][n - 2]])
-                if "Adj Low" in quotes.columns:
-                    quotes.loc[idx2, "Adj Low"] = _np.nanmin([quotes["Adj Low"][n - 1], quotes["Adj Low"][n - 2]])
                 if "Adj Close" in quotes.columns:
                     quotes.loc[idx2, "Adj Close"] = quotes["Adj Close"][n - 1]
                 quotes.loc[idx2, "Volume"] += quotes["Volume"][n - 1]
@@ -552,7 +589,7 @@ def safe_merge_dfs(df_main, df_sub, interval):
 
         df["_NewIndex"] = new_index
         # Duplicates present within periods but can aggregate
-        if data_col_name == "Dividends":
+        if data_col_name in ["Dividends", "Capital Gains"]:
             # Add
             df = df.groupby("_NewIndex").sum()
             df.index.name = None
@@ -690,7 +727,7 @@ def is_valid_timezone(tz: str) -> bool:
     return True
 
 
-def format_history_metadata(md):
+def format_history_metadata(md, tradingPeriodsOnly=True):
     if not isinstance(md, dict):
         return md
     if len(md) == 0:
@@ -698,60 +735,54 @@ def format_history_metadata(md):
 
     tz = md["exchangeTimezoneName"]
 
-    for k in ["firstTradeDate", "regularMarketTime"]:
-        if k in md and md[k] is not None:
-            md[k] = _pd.to_datetime(md[k], unit='s', utc=True).tz_convert(tz)
+    if not tradingPeriodsOnly:
+        for k in ["firstTradeDate", "regularMarketTime"]:
+            if k in md and md[k] is not None:
+                if isinstance(md[k], int):
+                    md[k] = _pd.to_datetime(md[k], unit='s', utc=True).tz_convert(tz)
 
-    if "currentTradingPeriod" in md:
-        for m in ["regular", "pre", "post"]:
-            if m in md["currentTradingPeriod"]:
-                for t in ["start", "end"]:
-                    md["currentTradingPeriod"][m][t] = \
-                        _pd.to_datetime(md["currentTradingPeriod"][m][t], unit='s', utc=True).tz_convert(tz)
-                del md["currentTradingPeriod"][m]["gmtoffset"]
-                del md["currentTradingPeriod"][m]["timezone"]
-
-    if "tradingPeriods" in md:
-        if md["tradingPeriods"] == {"pre":[], "post":[]}:
-            del md["tradingPeriods"]
+        if "currentTradingPeriod" in md:
+            for m in ["regular", "pre", "post"]:
+                if m in md["currentTradingPeriod"] and isinstance(md["currentTradingPeriod"][m]["start"], int):
+                    for t in ["start", "end"]:
+                        md["currentTradingPeriod"][m][t] = \
+                            _pd.to_datetime(md["currentTradingPeriod"][m][t], unit='s', utc=True).tz_convert(tz)
+                    del md["currentTradingPeriod"][m]["gmtoffset"]
+                    del md["currentTradingPeriod"][m]["timezone"]
 
     if "tradingPeriods" in md:
         tps = md["tradingPeriods"]
-        if isinstance(tps, list):
-            # Only regular times
-            regs_dict = [tps[i][0] for i in range(len(tps))]
-            pres_dict = None
-            posts_dict = None
-        elif isinstance(tps, dict):
-            # Includes pre- and post-market
-            pres_dict = [tps["pre"][i][0] for i in range(len(tps["pre"]))]
-            posts_dict = [tps["post"][i][0] for i in range(len(tps["post"]))]
-            regs_dict = [tps["regular"][i][0] for i in range(len(tps["regular"]))]
-        else:
-            raise Exception()
+        if tps == {"pre":[], "post":[]}:
+            # Ignore
+            pass
+        elif isinstance(tps, (list, dict)):
+            if isinstance(tps, list):
+                # Only regular times
+                df = _pd.DataFrame.from_records(_np.hstack(tps))
+                df = df.drop(["timezone", "gmtoffset"], axis=1)
+                df["start"] = _pd.to_datetime(df["start"], unit='s', utc=True).dt.tz_convert(tz)
+                df["end"] = _pd.to_datetime(df["end"], unit='s', utc=True).dt.tz_convert(tz)
+            elif isinstance(tps, dict):
+                # Includes pre- and post-market
+                pre_df = _pd.DataFrame.from_records(_np.hstack(tps["pre"]))
+                post_df = _pd.DataFrame.from_records(_np.hstack(tps["post"]))
+                regular_df = _pd.DataFrame.from_records(_np.hstack(tps["regular"]))
+
+                pre_df = pre_df.rename(columns={"start":"pre_start", "end":"pre_end"}).drop(["timezone", "gmtoffset"], axis=1)
+                post_df = post_df.rename(columns={"start":"post_start", "end":"post_end"}).drop(["timezone", "gmtoffset"], axis=1)
+                regular_df = regular_df.drop(["timezone", "gmtoffset"], axis=1)
+
+                cols = ["pre_start", "pre_end", "start", "end", "post_start", "post_end"]
+                df = regular_df.join(pre_df).join(post_df)
+                for c in cols:
+                    df[c] = _pd.to_datetime(df[c], unit='s', utc=True).dt.tz_convert(tz)
+                df = df[cols]
 
-        def _dict_to_table(d):
-            df = _pd.DataFrame.from_dict(d).drop(["timezone", "gmtoffset"], axis=1)
-            df["end"] = _pd.to_datetime(df["end"], unit='s', utc=True).dt.tz_convert(tz)
-            df["start"] = _pd.to_datetime(df["start"], unit='s', utc=True).dt.tz_convert(tz)
             df.index = _pd.to_datetime(df["start"].dt.date)
             df.index = df.index.tz_localize(tz)
-            return df
+            df.index.name = "Date"
 
-        df = _dict_to_table(regs_dict)
-        df_cols = ["start", "end"]
-        if pres_dict is not None:
-            pre_df = _dict_to_table(pres_dict)
-            df = df.merge(pre_df.rename(columns={"start":"pre_start", "end":"pre_end"}), left_index=True, right_index=True)
-            df_cols = ["pre_start", "pre_end"]+df_cols
-        if posts_dict is not None:
-            post_df = _dict_to_table(posts_dict)
-            df = df.merge(post_df.rename(columns={"start":"post_start", "end":"post_end"}), left_index=True, right_index=True)
-            df_cols = df_cols+["post_start", "post_end"]
-        df = df[df_cols]
-        df.index.name = "Date"
-
-        md["tradingPeriods"] = df
+            md["tradingPeriods"] = df
 
     return md
 
@@ -836,14 +867,21 @@ class _KVStore:
 
     def get(self, key: str) -> Union[str, None]:
         """Get value for key if it exists else returns None"""
-        item = self.conn.execute('select value from "kv" where key=?', (key,))
+        try:
+            item = self.conn.execute('select value from "kv" where key=?', (key,))
+        except _sqlite3.IntegrityError as e:
+            self.delete(key)
+            return None
         if item:
             return next(item, (None,))[0]
 
     def set(self, key: str, value: str) -> None:
-        with self._cache_mutex:
-            self.conn.execute('replace into "kv" (key, value) values (?,?)', (key, value))
-            self.conn.commit()
+        if value is None:
+            self.delete(key)
+        else:
+            with self._cache_mutex:
+                self.conn.execute('replace into "kv" (key, value) values (?,?)', (key, value))
+                self.conn.commit()
 
     def bulk_set(self, kvdata: Dict[str, str]):
         records = tuple(i for i in kvdata.items())
@@ -909,11 +947,23 @@ class _TzCache:
         if not _os.path.isfile(old_cache_file_path):
             return None
         try:
-            df = _pd.read_csv(old_cache_file_path, index_col="Ticker")
+            df = _pd.read_csv(old_cache_file_path, index_col="Ticker", on_bad_lines="skip")
         except _pd.errors.EmptyDataError:
             _os.remove(old_cache_file_path)
+        except TypeError:
+            _os.remove(old_cache_file_path)
         else:
-            self.tz_db.bulk_set(df.to_dict()['Tz'])
+            # Discard corrupt data:
+            df = df[~df["Tz"].isna().to_numpy()]
+            df = df[~(df["Tz"]=='').to_numpy()]
+            df = df[~df.index.isna()]
+            if not df.empty:
+                try:
+                    self.tz_db.bulk_set(df.to_dict()['Tz'])
+                except Exception as e:
+                    # Ignore
+                    pass
+
             _os.remove(old_cache_file_path)
 
 
@@ -944,9 +994,10 @@ def get_tz_cache():
             try:
                 _tz_cache = _TzCache()
             except _TzCacheException as err:
-                print("Failed to create TzCache, reason: {}".format(err))
-                print("TzCache will not be used.")
-                print("Tip: You can direct cache to use a different location with 'set_tz_cache_location(mylocation)'")
+                logger.error("Failed to create TzCache, reason: %s. "
+                             "TzCache will not be used. "
+                             "Tip: You can direct cache to use a different location with 'set_tz_cache_location(mylocation)'",
+                             err)
                 _tz_cache = _TzCacheDummy()
 
         return _tz_cache

yfinance/version.py

@@ -1 +1 @@
-version = "0.2.12"
+version = "0.2.19"