Bump version to 0.2.18

Fix handling Pandas parsing error during TZ-csv-cache migrate

.github/ISSUE_TEMPLATE/bug_report.md

@@ -7,7 +7,9 @@ assignees: ''
 
 ---
 
-# READ BEFORE POSTING
+# IMPORTANT
+
+If you want help, you got to read this first, follow the instructions.
 
 ### Are you up-to-date?
 
@@ -23,20 +25,19 @@ and comparing against [PIP](https://pypi.org/project/yfinance/#history).
 
 ### Does Yahoo actually have the data?
 
-Are spelling ticker *exactly* same as Yahoo?
+Are you spelling ticker *exactly* same as Yahoo?
 
-Visit `finance.yahoo.com` and confim they have your data. Maybe your ticker was delisted.
+Then visit `finance.yahoo.com` and confirm they have the data you want. Maybe your ticker was delisted, or your expectations of `yfinance` are wrong.
 
 ### Are you spamming Yahoo?
 
-Yahoo Finance free service has limit on query rate dependent on request - roughly 500/minute for prices, 10/minute for info. Them delaying or blocking your spam is not a bug.
+Yahoo Finance free service has rate-limiting depending on request type - roughly 60/minute for prices, 10/minute for info. Once limit hit, Yahoo can delay, block, or return bad data. Not a `yfinance` bug.
 
 ### Still think it's a bug?
 
-Delete this default message and submit your bug report here, providing the following as best you can:
+Delete this default message (all of it) and submit your bug report here, providing the following as best you can:
 
-- Simple code that reproduces your problem
-- Error message, with traceback if shown
-- Info about your system:
-  - yfinance version
-  - operating system
+- Simple code that reproduces your problem, that we can copy-paste-run
+- Exception message with full traceback, or proof `yfinance` returning bad data
+- `yfinance` version and Python version
+- Operating system type

CHANGELOG.rst

@@ -1,6 +1,56 @@
 Change Log
 ===========
 
+0.2.18
+------
+Fix 'fast_info' error '_np not found' #1496
+Fix bug in timezone cache #1498
+
+0.2.17
+------
+Fix prices error with Pandas 2.0 #1488
+
+0.2.16
+------
+Fix 'fast_info deprecated' msg appearing at Ticker() init
+
+0.2.15
+------
+Restore missing Ticker.info keys #1480
+
+0.2.14
+------
+Fix Ticker.info dict by fetching from API #1461
+
+0.2.13
+------
+Price bug fixes:
+- fetch big-interval with Capital Gains #1455
+- merging dividends & splits with prices #1452
+
+0.2.12
+------
+Disable annoying 'backup decrypt' msg
+
+0.2.11
+------
+Fix history_metadata accesses for unusual symbols #1411
+
+0.2.10
+------
+General
+- allow using sqlite3 < 3.8.2 #1380
+- add another backup decrypt option #1379
+Prices
+- restore original download() timezone handling #1385
+- fix & improve price repair #1289 2a2928b 86d6acc
+- drop intraday intervals if in post-market but prepost=False #1311
+Info
+- fast_info improvements:
+  - add camelCase keys, add dict functions values() & items() #1368
+  - fix fast_info["previousClose"] #1383
+- catch TypeError Exception #1397
+
 0.2.9
 -----
 - Fix fast_info bugs #1362

README.md

@@ -45,7 +45,7 @@ Yahoo! finance API is intended for personal use only.**
 ## News [2023-01-27]
 Since December 2022 Yahoo has been encrypting the web data that `yfinance` scrapes for non-market data. Fortunately the decryption keys are available, although Yahoo moved/changed them several times hence `yfinance` breaking several times. `yfinance` is now better prepared for any future changes by Yahoo.
 
-Why is Yahoo doing this? We don't know. Is it to stop scrapers? Maybe, so we've implemented changes to reduce load on Yahoo. In December we rolled out version 0.2 with optimised scraping. Then in 0.2.6 introduced `Ticker.fast_info`, providing much faster access to some `info` elements wherever possible e.g. price stats and forcing users to switch (sorry but we think necessary). `info` will continue to exist for as long as there are elements without a fast alternative.
+Why is Yahoo doing this? We don't know. Is it to stop scrapers? Maybe, so we've implemented changes to reduce load on Yahoo. In December we rolled out version 0.2 with optimised scraping. ~Then in 0.2.6 introduced `Ticker.fast_info`, providing much faster access to some `info` elements wherever possible e.g. price stats and forcing users to switch (sorry but we think necessary). `info` will continue to exist for as long as there are elements without a fast alternative.~ `info` now fixed and much faster than before.
 
 ## Quick Start
 
@@ -58,10 +58,8 @@ import yfinance as yf
 
 msft = yf.Ticker("MSFT")
 
-# get all stock info (slow)
+# get all stock info
 msft.info
-# fast access to subset of stock info (opportunistic)
-msft.fast_info
 
 # get historical market data
 hist = msft.history(period="1mo")
@@ -154,18 +152,7 @@ msft.option_chain(..., proxy="PROXY_SERVER")
 ...
 ```
 
-To use a custom `requests` session (for example to cache calls to the
-API or customize the `User-agent` header), pass a `session=` argument to
-the Ticker constructor.
-
-```python
-import requests_cache
-session = requests_cache.CachedSession('yfinance.cache')
-session.headers['User-agent'] = 'my-program/1.0'
-ticker = yf.Ticker('msft', session=session)
-# The scraped response will be stored in the cache
-ticker.actions
-```
+### Multiple tickers
 
 To initialize multiple `Ticker` objects, use
 
@@ -180,69 +167,54 @@ tickers.tickers['AAPL'].history(period="1mo")
 tickers.tickers['GOOG'].actions
 ```
 
-### Fetching data for multiple tickers
+To download price history into one table:
 
 ```python
 import yfinance as yf
 data = yf.download("SPY AAPL", start="2017-01-01", end="2017-04-30")
 ```
 
-I've also added some options to make life easier :)
+`yf.download()` and `Ticker.history()` have many options for configuring fetching and processing, e.g.:
 
 ```python
-data = yf.download(  # or pdr.get_data_yahoo(...
-        # tickers list or string as well
-        tickers = "SPY AAPL MSFT",
-
-        # use "period" instead of start/end
-        # valid periods: 1d,5d,1mo,3mo,6mo,1y,2y,5y,10y,ytd,max
-        # (optional, default is '1mo')
-        period = "ytd",
-
-        # fetch data by interval (including intraday if period < 60 days)
-        # valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
-        # (optional, default is '1d')
-        interval = "5d",
-
-        # Whether to ignore timezone when aligning ticker data from 
-        # different timezones. Default is False.
-        ignore_tz = False,
-
-        # group by ticker (to access via data['SPY'])
-        # (optional, default is 'column')
-        group_by = 'ticker',
-
-        # adjust all OHLC automatically
-        # (optional, default is False)
-        auto_adjust = True,
-
-        # attempt repair of Yahoo data issues
-        repair = False,
-
-        # download pre/post regular market hours data
-        # (optional, default is False)
-        prepost = True,
-
-        # use threads for mass downloading? (True/False/Integer)
-        # (optional, default is True)
-        threads = True,
-
-        # proxy URL scheme use use when downloading?
-        # (optional, default is None)
-        proxy = None
-    )
+yf.download(tickers = "SPY AAPL",  # list of tickers
+            period = "1y",         # time period
+            interval = "1d",       # trading interval
+            prepost = False,       # download pre/post market hours data?
+            repair = True)         # repair obvious price errors e.g. 100x?
 ```
 
-### Timezone cache store
+Review the [Wiki](https://github.com/ranaroussi/yfinance/wiki) for more options and detail.
+
+### Smarter scraping
+
+To use a custom `requests` session (for example to cache calls to the
+API or customize the `User-agent` header), pass a `session=` argument to
+the Ticker constructor.
 
-When fetching price data, all dates are localized to stock exchange timezone. 
-But timezone retrieval is relatively slow, so yfinance attemps to cache them 
-in your users cache folder. 
-You can direct cache to use a different location with `set_tz_cache_location()`:
 ```python
-import yfinance as yf
-yf.set_tz_cache_location("custom/cache/location")
-...
+import requests_cache
+session = requests_cache.CachedSession('yfinance.cache')
+session.headers['User-agent'] = 'my-program/1.0'
+ticker = yf.Ticker('msft', session=session)
+# The scraped response will be stored in the cache
+ticker.actions
+```
+
+Combine a `requests_cache` with rate-limiting to avoid triggering Yahoo's rate-limiter/blocker that can corrupt data.
+```python
+from requests import Session
+from requests_cache import CacheMixin, SQLiteCache
+from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
+from pyrate_limiter import Duration, RequestRate, Limiter
+class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
+    pass
+
+session = CachedLimiterSession(
+    limiter=Limiter(RequestRate(2, Duration.SECOND*5),  # max 2 requests per 5 seconds
+    bucket_class=MemoryQueueBucket,
+    backend=SQLiteCache("yfinance.cache"),
+)
 ```
 
 ### Managing Multi-Level Columns
@@ -260,9 +232,7 @@ yfinance?](https://stackoverflow.com/questions/63107801)
         -   How to download single or multiple tickers into a single
             dataframe with single level column names and a ticker column
 
----
-
-## `pandas_datareader` override
+### `pandas_datareader` override
 
 If your code uses `pandas_datareader` and you want to download data
 faster, you can "hijack" `pandas_datareader.data.get_data_yahoo()`
@@ -279,6 +249,18 @@ yf.pdr_override() # <== that's all it takes :-)
 data = pdr.get_data_yahoo("SPY", start="2017-01-01", end="2017-04-30")
 ```
 
+### Timezone cache store
+
+When fetching price data, all dates are localized to stock exchange timezone. 
+But timezone retrieval is relatively slow, so yfinance attemps to cache them 
+in your users cache folder. 
+You can direct cache to use a different location with `set_tz_cache_location()`:
+```python
+import yfinance as yf
+yf.set_tz_cache_location("custom/cache/location")
+...
+```
+
 ---
 
 ## Installation
@@ -306,11 +288,15 @@ To install `yfinance` using `conda`, see
 -   [html5lib](https://pypi.org/project/html5lib) \>= 1.1
 -   [cryptography](https://pypi.org/project/cryptography) \>= 3.3.2
 
-### Optional (if you want to use `pandas_datareader`)
+#### Optional (if you want to use `pandas_datareader`)
 
 -   [pandas\_datareader](https://github.com/pydata/pandas-datareader)
     \>= 0.4.0
 
+## Developers: want to contribute?
+
+`yfinance` relies on community to investigate bugs and contribute code. Developer guide: https://github.com/ranaroussi/yfinance/discussions/1084
+
 ---
 
 ### Legal Stuff

meta.yaml

@@ -1,5 +1,5 @@
 {% set name = "yfinance" %}
-{% set version = "0.2.9" %}
+{% set version = "0.2.18" %}
 
 package:
   name: "{{ name|lower }}"

tests/prices.py

@@ -230,6 +230,11 @@ class TestPriceHistory(unittest.TestCase):
                 print("{}-without-events missing these dates: {}".format(tkr, missing_from_df2))
                 raise
 
+    def test_monthlyWithEvents2(self):
+        # Simply check no exception from internal merge
+        tkr = "ABBV"
+        yf.Ticker("ABBV").history(period="max", interval="1mo")
+
     def test_tz_dst_ambiguous(self):
         # Reproduce issue #1100
         try:
@@ -381,6 +386,16 @@ class TestPriceHistory(unittest.TestCase):
         df = dat.history(start=start, interval="1wk")
         self.assertTrue((df.index.weekday == 0).all())
 
+    def test_aggregate_capital_gains(self):
+        # Setup
+        tkr = "FXAIX"
+        dat = yf.Ticker(tkr, session=self.session)
+        start = "2017-12-31"
+        end = "2019-12-31"
+        interval = "3mo"
+
+        df = dat.history(start=start, end=end, interval=interval)
+
 class TestPriceRepair(unittest.TestCase):
     session = None
 
@@ -426,16 +441,16 @@ class TestPriceRepair(unittest.TestCase):
                                  "Close":     [475,   473.5, 472,   473.5],
                                  "Adj Close": [475,   473.5, 472,   473.5],
                                  "Volume": [2295613, 2245604, 3000287, 2635611]},
-                                index=_pd.to_datetime([_dt.date(2022, 10, 23),
-                                                       _dt.date(2022, 10, 16),
-                                                       _dt.date(2022, 10, 9),
-                                                       _dt.date(2022, 10, 2)]))
+                                index=_pd.to_datetime([_dt.date(2022, 10, 24),
+                                                       _dt.date(2022, 10, 17),
+                                                       _dt.date(2022, 10, 10),
+                                                       _dt.date(2022, 10, 3)]))
         df = df.sort_index()
         df.index.name = "Date"
         df_bad = df.copy()
-        df_bad.loc["2022-10-23", "Close"] *= 100
-        df_bad.loc["2022-10-16", "Low"] *= 100
-        df_bad.loc["2022-10-2", "Open"] *= 100
+        df_bad.loc["2022-10-24", "Close"] *= 100
+        df_bad.loc["2022-10-17", "Low"] *= 100
+        df_bad.loc["2022-10-03", "Open"] *= 100
         df.index = df.index.tz_localize(tz_exchange)
         df_bad.index = df_bad.index.tz_localize(tz_exchange)
 

tests/ticker.py

@@ -52,12 +52,16 @@ class TestTicker(unittest.TestCase):
     def test_badTicker(self):
         # Check yfinance doesn't die when ticker delisted
 
-        tkr = "AM2Z.TA"
+        tkr = "DJI"  # typo of "^DJI"
         dat = yf.Ticker(tkr, session=self.session)
         dat.history(period="1wk")
         dat.history(start="2022-01-01")
         dat.history(start="2022-01-01", end="2022-03-01")
         yf.download([tkr], period="1wk")
+
+        for k in dat.fast_info:
+            dat.fast_info[k]
+
         dat.isin
         dat.major_holders
         dat.institutional_holders
@@ -91,43 +95,48 @@ class TestTicker(unittest.TestCase):
     def test_goodTicker(self):
         # that yfinance works when full api is called on same instance of ticker
 
-        tkr = "IBM"
-        dat = yf.Ticker(tkr, session=self.session)
+        tkrs = ["IBM"]
+        tkrs.append("QCSTIX")  # weird ticker, no price history but has previous close
+        for tkr in tkrs:
+            dat = yf.Ticker(tkr, session=self.session)
 
-        dat.isin
-        dat.major_holders
-        dat.institutional_holders
-        dat.mutualfund_holders
-        dat.dividends
-        dat.splits
-        dat.actions
-        dat.shares
-        dat.get_shares_full()
-        dat.info
-        dat.calendar
-        dat.recommendations
-        dat.earnings
-        dat.quarterly_earnings
-        dat.income_stmt
-        dat.quarterly_income_stmt
-        dat.balance_sheet
-        dat.quarterly_balance_sheet
-        dat.cashflow
-        dat.quarterly_cashflow
-        dat.recommendations_summary
-        dat.analyst_price_target
-        dat.revenue_forecasts
-        dat.sustainability
-        dat.options
-        dat.news
-        dat.earnings_trend
-        dat.earnings_dates
-        dat.earnings_forecasts
+            dat.history(period="1wk")
+            dat.history(start="2022-01-01")
+            dat.history(start="2022-01-01", end="2022-03-01")
+            yf.download([tkr], period="1wk")
 
-        dat.history(period="1wk")
-        dat.history(start="2022-01-01")
-        dat.history(start="2022-01-01", end="2022-03-01")
-        yf.download([tkr], period="1wk")
+            for k in dat.fast_info:
+                dat.fast_info[k]
+
+            dat.isin
+            dat.major_holders
+            dat.institutional_holders
+            dat.mutualfund_holders
+            dat.dividends
+            dat.splits
+            dat.actions
+            dat.shares
+            dat.get_shares_full()
+            dat.info
+            dat.calendar
+            dat.recommendations
+            dat.earnings
+            dat.quarterly_earnings
+            dat.income_stmt
+            dat.quarterly_income_stmt
+            dat.balance_sheet
+            dat.quarterly_balance_sheet
+            dat.cashflow
+            dat.quarterly_cashflow
+            dat.recommendations_summary
+            dat.analyst_price_target
+            dat.revenue_forecasts
+            dat.sustainability
+            dat.options
+            dat.news
+            dat.earnings_trend
+            dat.earnings_dates
+            dat.earnings_forecasts
 
 
 class TestTickerHistory(unittest.TestCase):
@@ -695,6 +704,11 @@ class TestTickerInfo(unittest.TestCase):
         self.assertEqual(self.symbols[0], data["symbol"], "Wrong symbol value in info dict")
 
     def test_fast_info(self):
+        f = yf.Ticker("AAPL", session=self.session).fast_info
+        for k in f:
+            self.assertIsNotNone(f[k])
+
+    def test_fast_info_matches_info(self):
         yf.scrapers.quote.PRUNE_INFO = False
 
         fast_info_keys = set()

yfinance/base.py

@@ -40,7 +40,7 @@ from . import shared
 from .scrapers.analysis import Analysis
 from .scrapers.fundamentals import Fundamentals
 from .scrapers.holders import Holders
-from .scrapers.quote import Quote
+from .scrapers.quote import Quote, FastInfo
 import json as _json
 
 _BASE_URL_ = 'https://query2.finance.yahoo.com'
@@ -48,448 +48,6 @@ _SCRAPE_URL_ = 'https://finance.yahoo.com/quote'
 _ROOT_URL_ = 'https://finance.yahoo.com'
 
 
-class FastInfo:
-    # Contain small subset of info[] items that can be fetched faster elsewhere.
-    # Imitates a dict.
-    def __init__(self, tickerBaseObject):
-        self._tkr = tickerBaseObject
-
-        self._prices_1y = None
-        self._prices_1wk_1h_prepost = None
-        self._prices_1wk_1h_reg = None
-        self._md = None
-
-        self._currency = None
-        self._quote_type = None
-        self._exchange = None
-        self._timezone = None
-
-        self._shares = None
-        self._mcap = None
-
-        self._open = None
-        self._day_high = None
-        self._day_low = None
-        self._last_price = None
-        self._last_volume = None
-
-        self._prev_close = None
-
-        self._reg_prev_close = None
-
-        self._50d_day_average = None
-        self._200d_day_average = None
-        self._year_high = None
-        self._year_low = None
-        self._year_change = None
-
-        self._10d_avg_vol = None
-        self._3mo_avg_vol = None
-
-        # attrs = utils.attributes(self)
-        # self.keys = attrs.keys()
-        # utils.attributes is calling each method, bad! Have to hardcode
-        _properties = ["currency", "quote_type", "exchange", "timezone"]
-        _properties += ["shares", "market_cap"]
-        _properties += ["last_price", "previous_close", "open", "day_high", "day_low"]
-        _properties += ["regular_market_previous_close"]
-        _properties += ["last_volume"]
-        _properties += ["fifty_day_average", "two_hundred_day_average", "ten_day_average_volume", "three_month_average_volume"]
-        _properties += ["year_high", "year_low", "year_change"]
-
-        # Because released before fixing key case, need to officially support 
-        # camel-case but also secretly support snake-case
-        base_keys = [k for k in _properties if not '_' in k]
-
-        sc_keys = [k for k in _properties if '_' in k]
-
-        self._sc_to_cc_key = {k:utils.snake_case_2_camelCase(k) for k in sc_keys}
-        self._cc_to_sc_key = {v:k for k,v in self._sc_to_cc_key.items()}
- 
-        self._public_keys = sorted(base_keys + list(self._sc_to_cc_key.values()))
-        self._keys = sorted(self._public_keys + sc_keys)
-
-    # dict imitation:
-    def keys(self):
-        return self._public_keys
-    def items(self):
-        return [(k,self[k]) for k in self._public_keys]
-    def values(self):
-        return [self[k] for k in self._public_keys]
-    def get(self, key, default=None):
-        if key in self.keys():
-            if key in self._cc_to_sc_key:
-                key = self._cc_to_sc_key[key]
-            return self[key]
-        return default
-    def __getitem__(self, k):
-        if not isinstance(k, str):
-            raise KeyError(f"key must be a string")
-        if not k in self._keys:
-            raise KeyError(f"'{k}' not valid key. Examine 'FastInfo.keys()'")
-        if k in self._cc_to_sc_key:
-            k = self._cc_to_sc_key[k]
-        return getattr(self, k)
-    def __contains__(self, k):
-        return k in self.keys()
-    def __iter__(self):
-        return iter(self.keys())
-
-    def __str__(self):
-        return "lazy-loading dict with keys = " + str(self.keys())
-    def __repr__(self):
-        return self.__str__()
-
-    def toJSON(self, indent=4):
-        d = {k:self[k] for k in self.keys()}
-        return _json.dumps({k:self[k] for k in self.keys()}, indent=indent)
-
-    def _get_1y_prices(self, fullDaysOnly=False):
-        if self._prices_1y is None:
-            self._prices_1y = self._tkr.history(period="380d", auto_adjust=False, debug=False, keepna=True)
-            self._md = self._tkr.get_history_metadata()
-            try:
-                ctp = self._md["currentTradingPeriod"]
-                self._today_open = pd.to_datetime(ctp["regular"]["start"], unit='s', utc=True).tz_convert(self.timezone)
-                self._today_close = pd.to_datetime(ctp["regular"]["end"], unit='s', utc=True).tz_convert(self.timezone)
-                self._today_midnight = self._today_close.ceil("D")
-            except:
-                self._today_open = None
-                self._today_close = None
-                self._today_midnight = None
-                raise
-
-        if self._prices_1y.empty:
-            return self._prices_1y
-
-        dnow = pd.Timestamp.utcnow().tz_convert(self.timezone).date()
-        d1 = dnow
-        d0 = (d1 + _datetime.timedelta(days=1)) - utils._interval_to_timedelta("1y")
-        if fullDaysOnly and self._exchange_open_now():
-            # Exclude today
-            d1 -= utils._interval_to_timedelta("1d")
-        return self._prices_1y.loc[str(d0):str(d1)]
-
-    def _get_1wk_1h_prepost_prices(self):
-        if self._prices_1wk_1h_prepost is None:
-            self._prices_1wk_1h_prepost = self._tkr.history(period="1wk", interval="1h", auto_adjust=False, prepost=True, debug=False)
-        return self._prices_1wk_1h_prepost
-
-    def _get_1wk_1h_reg_prices(self):
-        if self._prices_1wk_1h_reg is None:
-            self._prices_1wk_1h_reg = self._tkr.history(period="1wk", interval="1h", auto_adjust=False, prepost=False, debug=False)
-        return self._prices_1wk_1h_reg
-
-    def _get_exchange_metadata(self):
-        if self._md is not None:
-            return self._md
-
-        self._get_1y_prices()
-        self._md = self._tkr.get_history_metadata()
-        return self._md
-
-    def _exchange_open_now(self):
-        t = pd.Timestamp.utcnow()
-        self._get_exchange_metadata()
-
-        # if self._today_open is None and self._today_close is None:
-        #     r = False
-        # else:
-        #     r = self._today_open <= t and t < self._today_close
-
-        # if self._today_midnight is None:
-        #     r = False
-        # elif self._today_midnight.date() > t.tz_convert(self.timezone).date():
-        #     r = False
-        # else:
-        #     r = t < self._today_midnight
-
-        last_day_cutoff = self._get_1y_prices().index[-1] + _datetime.timedelta(days=1)
-        last_day_cutoff += _datetime.timedelta(minutes=20)
-        r = t < last_day_cutoff
-
-        # print("_exchange_open_now() returning", r)
-        return r
-
-    @property
-    def currency(self):
-        if self._currency is not None:
-            return self._currency
-
-        if self._tkr._history_metadata is None:
-            self._get_1y_prices()
-        md = self._tkr.get_history_metadata()
-        self._currency = md["currency"]
-        return self._currency
-
-    @property
-    def quote_type(self):
-        if self._quote_type is not None:
-            return self._quote_type
-
-        if self._tkr._history_metadata is None:
-            self._get_1y_prices()
-        md = self._tkr.get_history_metadata()
-        self._quote_type = md["instrumentType"]
-        return self._quote_type
-
-    @property
-    def exchange(self):
-        if self._exchange is not None:
-            return self._exchange
-
-        self._exchange = self._get_exchange_metadata()["exchangeName"]
-        return self._exchange
-
-    @property
-    def timezone(self):
-        if self._timezone is not None:
-            return self._timezone
-
-        self._timezone = self._get_exchange_metadata()["exchangeTimezoneName"]
-        return self._timezone
-
-    @property
-    def shares(self):
-        if self._shares is not None:
-            return self._shares
-
-        shares = self._tkr.get_shares_full(start=pd.Timestamp.utcnow().date()-pd.Timedelta(days=548))
-        if shares is None:
-            # Requesting 18 months failed, so fallback to shares which should include last year
-            shares = self._tkr.get_shares()
-        if shares is not None:
-            if isinstance(shares, pd.DataFrame):
-                shares = shares[shares.columns[0]]
-            self._shares = int(shares.iloc[-1])
-        return self._shares
-
-    @property
-    def last_price(self):
-        if self._last_price is not None:
-            return self._last_price
-        prices = self._get_1y_prices()
-        if prices.empty:
-            self._last_price = self._get_exchange_metadata()["regularMarketPrice"]
-        else:
-            self._last_price = float(prices["Close"].iloc[-1])
-            if _np.isnan(self._last_price):
-                self._last_price = self._get_exchange_metadata()["regularMarketPrice"]
-        return self._last_price
-
-    @property
-    def previous_close(self):
-        if self._prev_close is not None:
-            return self._prev_close
-        prices = self._get_1wk_1h_prepost_prices()
-        prices = prices[["Close"]].groupby(prices.index.date).last()
-        if prices.shape[0] < 2:
-            # Very few symbols have previousClose despite no 
-            # no trading data. E.g. 'QCSTIX'.
-            # So fallback to original info[] if available.
-            self._tkr.info  # trigger fetch
-            if "previousClose" in self._tkr._quote._retired_info:
-                self._prev_close = self._tkr._quote._retired_info["previousClose"]
-        else:
-            self._prev_close = float(prices["Close"].iloc[-2])
-        return self._prev_close
-
-    @property
-    def regular_market_previous_close(self):
-        if self._reg_prev_close is not None:
-            return self._reg_prev_close
-        prices = self._get_1y_prices()
-        if prices.shape[0] == 1:
-            # Tiny % of tickers don't return daily history before last trading day, 
-            # so backup option is hourly history:
-            prices = self._get_1wk_1h_reg_prices()
-            prices = prices[["Close"]].groupby(prices.index.date).last()
-        if prices.shape[0] < 2:
-            # Very few symbols have regularMarketPreviousClose despite no 
-            # no trading data. E.g. 'QCSTIX'.
-            # So fallback to original info[] if available.
-            self._tkr.info  # trigger fetch
-            if "regularMarketPreviousClose" in self._tkr._quote._retired_info:
-                self._reg_prev_close = self._tkr._quote._retired_info["regularMarketPreviousClose"]
-        else:
-            self._reg_prev_close = float(prices["Close"].iloc[-2])
-        return self._reg_prev_close
-
-    @property
-    def open(self):
-        if self._open is not None:
-            return self._open
-        prices = self._get_1y_prices()
-        if prices.empty:
-            self._open = None
-        else:
-            self._open = float(prices["Open"].iloc[-1])
-            if _np.isnan(self._open):
-                self._open = None
-        return self._open
-
-    @property
-    def day_high(self):
-        if self._day_high is not None:
-            return self._day_high
-        prices = self._get_1y_prices()
-        if prices.empty:
-            self._day_high = None
-        else:
-            self._day_high = float(prices["High"].iloc[-1])
-            if _np.isnan(self._day_high):
-                self._day_high = None
-        return self._day_high
-
-    @property
-    def day_low(self):
-        if self._day_low is not None:
-            return self._day_low
-        prices = self._get_1y_prices()
-        if prices.empty:
-            self._day_low = None
-        else:
-            self._day_low = float(prices["Low"].iloc[-1])
-            if _np.isnan(self._day_low):
-                self._day_low = None
-        return self._day_low
-
-    @property
-    def last_volume(self):
-        if self._last_volume is not None:
-            return self._last_volume
-        prices = self._get_1y_prices()
-        self._last_volume = None if prices.empty else int(prices["Volume"].iloc[-1])
-        return self._last_volume
-
-    @property
-    def fifty_day_average(self):
-        if self._50d_day_average is not None:
-            return self._50d_day_average
-
-        prices = self._get_1y_prices(fullDaysOnly=True)
-        if prices.empty:
-            self._50d_day_average = None
-        else:
-            n = prices.shape[0]
-            a = n-50
-            b = n
-            if a < 0:
-                a = 0
-            self._50d_day_average = float(prices["Close"].iloc[a:b].mean())
-
-        return self._50d_day_average
-
-    @property
-    def two_hundred_day_average(self):
-        if self._200d_day_average is not None:
-            return self._200d_day_average
-
-        prices = self._get_1y_prices(fullDaysOnly=True)
-        if prices.empty:
-            self._200d_day_average = None
-        else:
-            n = prices.shape[0]
-            a = n-200
-            b = n
-            if a < 0:
-                a = 0
-
-            self._200d_day_average = float(prices["Close"].iloc[a:b].mean())
-
-        return self._200d_day_average
-
-    @property
-    def ten_day_average_volume(self):
-        if self._10d_avg_vol is not None:
-            return self._10d_avg_vol
-
-        prices = self._get_1y_prices(fullDaysOnly=True)
-        if prices.empty:
-            self._10d_avg_vol = None
-        else:
-            n = prices.shape[0]
-            a = n-10
-            b = n
-            if a < 0:
-                a = 0
-            self._10d_avg_vol = int(prices["Volume"].iloc[a:b].mean())
-
-        return self._10d_avg_vol
-
-    @property
-    def three_month_average_volume(self):
-        if self._3mo_avg_vol is not None:
-            return self._3mo_avg_vol
-
-        prices = self._get_1y_prices(fullDaysOnly=True)
-        if prices.empty:
-            self._3mo_avg_vol = None
-        else:
-            dt1 = prices.index[-1]
-            dt0 = dt1 - utils._interval_to_timedelta("3mo") + utils._interval_to_timedelta("1d")
-            self._3mo_avg_vol = int(prices.loc[dt0:dt1, "Volume"].mean())
-
-        return self._3mo_avg_vol
-
-    @property
-    def year_high(self):
-        if self._year_high is not None:
-            return self._year_high
-
-        prices = self._get_1y_prices(fullDaysOnly=True)
-        if prices.empty:
-            prices = self._get_1y_prices(fullDaysOnly=False)
-        self._year_high = float(prices["High"].max())
-        return self._year_high
-
-    @property
-    def year_low(self):
-        if self._year_low is not None:
-            return self._year_low
-
-        prices = self._get_1y_prices(fullDaysOnly=True)
-        if prices.empty:
-            prices = self._get_1y_prices(fullDaysOnly=False)
-        self._year_low = float(prices["Low"].min())
-        return self._year_low
-
-    @property
-    def year_change(self):
-        if self._year_change is not None:
-            return self._year_change
-
-        prices = self._get_1y_prices(fullDaysOnly=True)
-        if prices.shape[0] >= 2:
-            self._year_change = (prices["Close"].iloc[-1] - prices["Close"].iloc[0]) / prices["Close"].iloc[0]
-            self._year_change = float(self._year_change)
-        return self._year_change
-
-    @property
-    def market_cap(self):
-        if self._mcap is not None:
-            return self._mcap
-
-        try:
-            shares = self.shares
-        except Exception as e:
-            if "Cannot retrieve share count" in str(e):
-                shares = None
-            else:
-                raise
-
-        if shares is None:
-            # Very few symbols have marketCap despite no share count.
-            # E.g. 'BTC-USD'
-            # So fallback to original info[] if available.
-            self._tkr.info
-            if "marketCap" in self._tkr._quote._retired_info:
-                self._mcap = self._tkr._quote._retired_info["marketCap"]
-        else:
-            self._mcap = float(shares * self.last_price)
-        return self._mcap
-
-
 class TickerBase:
     def __init__(self, ticker, session=None):
         self.ticker = ticker.upper()
@@ -520,7 +78,7 @@ class TickerBase:
         self._quote = Quote(self._data)
         self._fundamentals = Fundamentals(self._data)
 
-        self._fast_info = FastInfo(self)
+        self._fast_info = None
 
     def stats(self, proxy=None):
         ticker_url = "{}/{}".format(self._scrape_url, self.ticker)
@@ -543,11 +101,13 @@ class TickerBase:
                 Valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
                 Intraday data cannot extend last 60 days
             start: str
-                Download start date string (YYYY-MM-DD) or _datetime.
+                Download start date string (YYYY-MM-DD) or _datetime, inclusive.
                 Default is 1900-01-01
+                E.g. for start="2020-01-01", the first data point will be on "2020-01-01"
             end: str
-                Download end date string (YYYY-MM-DD) or _datetime.
+                Download end date string (YYYY-MM-DD) or _datetime, exclusive.
                 Default is now
+                E.g. for end="2023-01-01", the last data point will be on "2022-12-31"
             prepost : bool
                 Include Pre and Post market data in results?
                 Default is False
@@ -636,7 +196,7 @@ class TickerBase:
             get_fn = self._data.get
             if end is not None:
                 end_dt = _pd.Timestamp(end, unit='s').tz_localize("UTC")
-                dt_now = end_dt.tzinfo.localize(_datetime.datetime.utcnow())
+                dt_now = _pd.Timestamp.utcnow()
                 data_delay = _datetime.timedelta(minutes=30)
                 if end_dt+data_delay <= dt_now:
                     # Date range in past so safe to fetch through cache:
@@ -746,30 +306,28 @@ class TickerBase:
         if not expect_capital_gains:
             capital_gains = None
 
-        if start is not None:
-            # Note: use pandas Timestamp as datetime.utcfromtimestamp has bugs on windows
-            # https://github.com/python/cpython/issues/81708
-            startDt = _pd.Timestamp(start, unit='s')
-            if dividends is not None:
-                dividends = dividends[dividends.index>=startDt]
-            if capital_gains is not None:
-                capital_gains = capital_gains[capital_gains.index>=startDt]
-            if splits is not None:
-                splits = splits[splits.index >= startDt]
-        if end is not None:
-            endDt = _pd.Timestamp(end, unit='s')
-            if dividends is not None:
-                dividends = dividends[dividends.index<endDt]
-            if capital_gains is not None:
-                capital_gains = capital_gains[capital_gains.index<endDt]
-            if splits is not None:
-                splits = splits[splits.index < endDt]
         if splits is not None:
             splits = utils.set_df_tz(splits, interval, tz_exchange)
         if dividends is not None:
             dividends = utils.set_df_tz(dividends, interval, tz_exchange)
         if capital_gains is not None:
             capital_gains = utils.set_df_tz(capital_gains, interval, tz_exchange)
+        if start is not None:
+            startDt = quotes.index[0].floor('D')
+            if dividends is not None:
+                dividends = dividends.loc[startDt:]
+            if capital_gains is not None:
+                capital_gains = capital_gains.loc[startDt:]
+            if splits is not None:
+                splits = splits.loc[startDt:]
+        if end is not None:
+            endDt = _pd.Timestamp(end, unit='s').tz_localize(tz)
+            if dividends is not None:
+                dividends = dividends[dividends.index < endDt]
+            if capital_gains is not None:
+                capital_gains = capital_gains[capital_gains.index < endDt]
+            if splits is not None:
+                splits = splits[splits.index < endDt]
 
         # Prepare for combine
         intraday = params["interval"][-1] in ("m", 'h')
@@ -1024,6 +582,8 @@ class TickerBase:
                 if not silent:
                     print("YF: WARNING: Cannot reconstruct because Yahoo not returning data in interval")
                 continue
+            # Discard the buffer
+            df_fine = df_fine.loc[g[0] : g[-1]+itds[sub_interval]-_datetime.timedelta(milliseconds=1)]
 
             df_fine["ctr"] = 0
             if interval == "1wk":
@@ -1041,7 +601,8 @@ class TickerBase:
                 grp_col = "intervalID"
             df_fine = df_fine[~df_fine[price_cols].isna().all(axis=1)]
 
-            df_new = df_fine.groupby(grp_col).agg(
+            df_fine_grp = df_fine.groupby(grp_col)
+            df_new = df_fine_grp.agg(
                 Open=("Open", "first"),
                 Close=("Close", "last"),
                 AdjClose=("Adj Close", "last"),
@@ -1082,7 +643,13 @@ class TickerBase:
                     df_block_calib[f,j] = 1
                     df_new_calib[f,j] = 1
             ratios = df_block_calib[calib_filter] / df_new_calib[calib_filter]
-            ratio = _np.mean(ratios)
+            weights = df_fine_grp.size()
+            weights.index = df_new.index
+            weights = weights[weights.index.isin(common_index)].to_numpy().astype(float)
+            weights = weights[:,None]  # transpose
+            weights = _np.tile(weights, len(price_cols))  # 1D -> 2D
+            weights = weights[calib_filter]  # flatten
+            ratio = _np.average(ratios, weights=weights)
             if debug:
                 print(f"- price calibration ratio (raw) = {ratio}")
             ratio_rcp = round(1.0 / ratio, 1)
@@ -1455,6 +1022,8 @@ class TickerBase:
 
     @property
     def fast_info(self):
+        if self._fast_info is None:
+            self._fast_info = FastInfo(self)
         return self._fast_info
 
     @property

yfinance/data.py

@@ -15,6 +15,8 @@ else:
 import requests as requests
 import re
 from bs4 import BeautifulSoup
+import random
+import time
 
 from frozendict import frozendict
 
@@ -60,7 +62,7 @@ def _extract_extra_keys_from_stores(data):
             new_keys_uniq.append(k)
             new_keys_uniq_values.add(v)
 
-    return new_keys_uniq
+    return [data[k] for k in new_keys_uniq]
 
 
 def decrypt_cryptojs_aes_stores(data, keys=None):
@@ -202,6 +204,11 @@ class TickerData:
             proxy = {"https": proxy}
         return proxy
 
+    def get_raw_json(self, url, user_agent_headers=None, params=None, proxy=None, timeout=30):
+        response = self.get(url, user_agent_headers=user_agent_headers, params=params, proxy=proxy, timeout=timeout)
+        response.raise_for_status()
+        return response.json()
+
     def _get_decryption_keys_from_yahoo_js(self, soup):
         result = None
 
@@ -250,15 +257,16 @@ class TickerData:
                 response_js.close()
             if len(re_keys) == key_count:
                 break
-        re_obj = {}
-        missing_val = False
-        for k in re_keys:
-            if not re_data.get(k):
-                missing_val = True
-                break
-            re_obj.update({k: re_data.get(k)})
-        if not missing_val:
-            return [''.join(re_obj.values())]
+        if len(re_keys) > 0:
+            re_obj = {}
+            missing_val = False
+            for k in re_keys:
+                if not re_data.get(k):
+                    missing_val = True
+                    break
+                re_obj.update({k: re_data.get(k)})
+            if not missing_val:
+                return [''.join(re_obj.values())]
 
         return []
 
@@ -289,19 +297,22 @@ class TickerData:
         # Gather decryption keys:
         soup = BeautifulSoup(response.content, "html.parser")
         keys = self._get_decryption_keys_from_yahoo_js(soup)
+        # if len(keys) == 0:
+        #     msg = "No decryption keys could be extracted from JS file."
+        #     if "requests_cache" in str(type(response)):
+        #         msg += " Try flushing your 'requests_cache', probably parsing old JS."
+        #     print("WARNING: " + msg + " Falling back to backup decrypt methods.")
         if len(keys) == 0:
-            msg = "No decryption keys could be extracted from JS file."
-            if "requests_cache" in str(type(response)):
-                msg += " Try flushing your 'requests_cache', probably parsing old JS."
-            print("WARNING: " + msg + " Falling back to backup decrypt methods.")
-        if len(keys) == 0:
+            keys = []
+            try:
+                extra_keys = _extract_extra_keys_from_stores(data)
+                keys = [''.join(extra_keys[-4:])]
+            except:
+                pass
+            #
             keys_url = "https://github.com/ranaroussi/yfinance/raw/main/yfinance/scrapers/yahoo-keys.txt"
             response_gh = self.cache_get(keys_url)
-            keys = response_gh.text.splitlines()
-            extra_keys = _extract_extra_keys_from_stores(data)
-            if len(extra_keys) < 10:
-                # Only brute-force with these extra keys if few
-                keys += extra_keys
+            keys += response_gh.text.splitlines()
 
         # Decrypt!
         stores = decrypt_cryptojs_aes_stores(data, keys)

yfinance/multi.py

@@ -29,7 +29,7 @@ from . import Ticker, utils
 from . import shared
 
 
-def download(tickers, start=None, end=None, actions=False, threads=True, ignore_tz=False,
+def download(tickers, start=None, end=None, actions=False, threads=True, ignore_tz=None,
              group_by='column', auto_adjust=False, back_adjust=False, repair=False, keepna=False,
              progress=True, period="max", show_errors=True, interval="1d", prepost=False,
              proxy=None, rounding=False, timeout=10):
@@ -44,11 +44,13 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
             Valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
             Intraday data cannot extend last 60 days
         start: str
-            Download start date string (YYYY-MM-DD) or _datetime.
+            Download start date string (YYYY-MM-DD) or _datetime, inclusive.
             Default is 1900-01-01
+            E.g. for start="2020-01-01", the first data point will be on "2020-01-01"
         end: str
-            Download end date string (YYYY-MM-DD) or _datetime.
+            Download end date string (YYYY-MM-DD) or _datetime, exclusive.
             Default is now
+            E.g. for end="2023-01-01", the last data point will be on "2022-12-31"
         group_by : str
             Group by 'ticker' or 'column' (default)
         prepost : bool
@@ -68,7 +70,7 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
             How many threads to use for mass downloading. Default is True
         ignore_tz: bool
             When combining from different timezones, ignore that part of datetime.
-            Default is False
+            Default depends on interval. Intraday = False. Day+ = True.
         proxy: str
             Optional. Proxy server URL scheme. Default is None
         rounding: bool
@@ -80,6 +82,14 @@ def download(tickers, start=None, end=None, actions=False, threads=True, ignore_
             seconds. (Can also be a fraction of a second e.g. 0.01)
     """
 
+    if ignore_tz is None:
+        # Set default value depending on interval
+        if interval[1:] in ['m', 'h']:
+            # Intraday
+            ignore_tz = False
+        else:
+            ignore_tz = True
+
     # create ticker list
     tickers = tickers if isinstance(
         tickers, (list, set, tuple)) else tickers.replace(',', ' ').split()

yfinance/scrapers/quote.py

@@ -2,6 +2,7 @@ import datetime
 import json
 
 import pandas as pd
+import numpy as _np
 
 from yfinance import utils
 from yfinance.data import TickerData
@@ -19,6 +20,7 @@ info_retired_keys = info_retired_keys_price | info_retired_keys_exchange | info_
 
 PRUNE_INFO = True
 # PRUNE_INFO = False
+_BASIC_URL_ = "https://query2.finance.yahoo.com/v10/finance/quoteSummary"
 
 
 from collections.abc import MutableMapping
@@ -73,6 +75,462 @@ class InfoDictWrapper(MutableMapping):
         return k
 
 
+class FastInfo:
+    # Contain small subset of info[] items that can be fetched faster elsewhere.
+    # Imitates a dict.
+    def __init__(self, tickerBaseObject):
+        utils.print_once("yfinance: Note: 'Ticker.info' dict is now fixed & improved, 'fast_info' is no longer faster")
+
+        self._tkr = tickerBaseObject
+
+        self._prices_1y = None
+        self._prices_1wk_1h_prepost = None
+        self._prices_1wk_1h_reg = None
+        self._md = None
+
+        self._currency = None
+        self._quote_type = None
+        self._exchange = None
+        self._timezone = None
+
+        self._shares = None
+        self._mcap = None
+
+        self._open = None
+        self._day_high = None
+        self._day_low = None
+        self._last_price = None
+        self._last_volume = None
+
+        self._prev_close = None
+
+        self._reg_prev_close = None
+
+        self._50d_day_average = None
+        self._200d_day_average = None
+        self._year_high = None
+        self._year_low = None
+        self._year_change = None
+
+        self._10d_avg_vol = None
+        self._3mo_avg_vol = None
+
+        # attrs = utils.attributes(self)
+        # self.keys = attrs.keys()
+        # utils.attributes is calling each method, bad! Have to hardcode
+        _properties = ["currency", "quote_type", "exchange", "timezone"]
+        _properties += ["shares", "market_cap"]
+        _properties += ["last_price", "previous_close", "open", "day_high", "day_low"]
+        _properties += ["regular_market_previous_close"]
+        _properties += ["last_volume"]
+        _properties += ["fifty_day_average", "two_hundred_day_average", "ten_day_average_volume", "three_month_average_volume"]
+        _properties += ["year_high", "year_low", "year_change"]
+
+        # Because released before fixing key case, need to officially support 
+        # camel-case but also secretly support snake-case
+        base_keys = [k for k in _properties if not '_' in k]
+
+        sc_keys = [k for k in _properties if '_' in k]
+
+        self._sc_to_cc_key = {k:utils.snake_case_2_camelCase(k) for k in sc_keys}
+        self._cc_to_sc_key = {v:k for k,v in self._sc_to_cc_key.items()}
+ 
+        self._public_keys = sorted(base_keys + list(self._sc_to_cc_key.values()))
+        self._keys = sorted(self._public_keys + sc_keys)
+
+    # dict imitation:
+    def keys(self):
+        return self._public_keys
+    def items(self):
+        return [(k,self[k]) for k in self._public_keys]
+    def values(self):
+        return [self[k] for k in self._public_keys]
+    def get(self, key, default=None):
+        if key in self.keys():
+            if key in self._cc_to_sc_key:
+                key = self._cc_to_sc_key[key]
+            return self[key]
+        return default
+    def __getitem__(self, k):
+        if not isinstance(k, str):
+            raise KeyError(f"key must be a string")
+        if not k in self._keys:
+            raise KeyError(f"'{k}' not valid key. Examine 'FastInfo.keys()'")
+        if k in self._cc_to_sc_key:
+            k = self._cc_to_sc_key[k]
+        return getattr(self, k)
+    def __contains__(self, k):
+        return k in self.keys()
+    def __iter__(self):
+        return iter(self.keys())
+
+    def __str__(self):
+        return "lazy-loading dict with keys = " + str(self.keys())
+    def __repr__(self):
+        return self.__str__()
+
+    def toJSON(self, indent=4):
+        d = {k:self[k] for k in self.keys()}
+        return _json.dumps({k:self[k] for k in self.keys()}, indent=indent)
+
+    def _get_1y_prices(self, fullDaysOnly=False):
+        if self._prices_1y is None:
+            self._prices_1y = self._tkr.history(period="380d", auto_adjust=False, debug=False, keepna=True)
+            self._md = self._tkr.get_history_metadata()
+            try:
+                ctp = self._md["currentTradingPeriod"]
+                self._today_open = pd.to_datetime(ctp["regular"]["start"], unit='s', utc=True).tz_convert(self.timezone)
+                self._today_close = pd.to_datetime(ctp["regular"]["end"], unit='s', utc=True).tz_convert(self.timezone)
+                self._today_midnight = self._today_close.ceil("D")
+            except:
+                self._today_open = None
+                self._today_close = None
+                self._today_midnight = None
+                raise
+
+        if self._prices_1y.empty:
+            return self._prices_1y
+
+        dnow = pd.Timestamp.utcnow().tz_convert(self.timezone).date()
+        d1 = dnow
+        d0 = (d1 + datetime.timedelta(days=1)) - utils._interval_to_timedelta("1y")
+        if fullDaysOnly and self._exchange_open_now():
+            # Exclude today
+            d1 -= utils._interval_to_timedelta("1d")
+        return self._prices_1y.loc[str(d0):str(d1)]
+
+    def _get_1wk_1h_prepost_prices(self):
+        if self._prices_1wk_1h_prepost is None:
+            self._prices_1wk_1h_prepost = self._tkr.history(period="1wk", interval="1h", auto_adjust=False, prepost=True, debug=False)
+        return self._prices_1wk_1h_prepost
+
+    def _get_1wk_1h_reg_prices(self):
+        if self._prices_1wk_1h_reg is None:
+            self._prices_1wk_1h_reg = self._tkr.history(period="1wk", interval="1h", auto_adjust=False, prepost=False, debug=False)
+        return self._prices_1wk_1h_reg
+
+    def _get_exchange_metadata(self):
+        if self._md is not None:
+            return self._md
+
+        self._get_1y_prices()
+        self._md = self._tkr.get_history_metadata()
+        return self._md
+
+    def _exchange_open_now(self):
+        t = pd.Timestamp.utcnow()
+        self._get_exchange_metadata()
+
+        # if self._today_open is None and self._today_close is None:
+        #     r = False
+        # else:
+        #     r = self._today_open <= t and t < self._today_close
+
+        # if self._today_midnight is None:
+        #     r = False
+        # elif self._today_midnight.date() > t.tz_convert(self.timezone).date():
+        #     r = False
+        # else:
+        #     r = t < self._today_midnight
+
+        last_day_cutoff = self._get_1y_prices().index[-1] + datetime.timedelta(days=1)
+        last_day_cutoff += datetime.timedelta(minutes=20)
+        r = t < last_day_cutoff
+
+        # print("_exchange_open_now() returning", r)
+        return r
+
+    @property
+    def currency(self):
+        if self._currency is not None:
+            return self._currency
+
+        if self._tkr._history_metadata is None:
+            self._get_1y_prices()
+        md = self._tkr.get_history_metadata()
+        self._currency = md["currency"]
+        return self._currency
+
+    @property
+    def quote_type(self):
+        if self._quote_type is not None:
+            return self._quote_type
+
+        if self._tkr._history_metadata is None:
+            self._get_1y_prices()
+        md = self._tkr.get_history_metadata()
+        self._quote_type = md["instrumentType"]
+        return self._quote_type
+
+    @property
+    def exchange(self):
+        if self._exchange is not None:
+            return self._exchange
+
+        self._exchange = self._get_exchange_metadata()["exchangeName"]
+        return self._exchange
+
+    @property
+    def timezone(self):
+        if self._timezone is not None:
+            return self._timezone
+
+        self._timezone = self._get_exchange_metadata()["exchangeTimezoneName"]
+        return self._timezone
+
+    @property
+    def shares(self):
+        if self._shares is not None:
+            return self._shares
+
+        shares = self._tkr.get_shares_full(start=pd.Timestamp.utcnow().date()-pd.Timedelta(days=548))
+        if shares is None:
+            # Requesting 18 months failed, so fallback to shares which should include last year
+            shares = self._tkr.get_shares()
+        if shares is not None:
+            if isinstance(shares, pd.DataFrame):
+                shares = shares[shares.columns[0]]
+            self._shares = int(shares.iloc[-1])
+        return self._shares
+
+    @property
+    def last_price(self):
+        if self._last_price is not None:
+            return self._last_price
+        prices = self._get_1y_prices()
+        if prices.empty:
+            md = self._get_exchange_metadata()
+            if "regularMarketPrice" in md:
+                self._last_price = md["regularMarketPrice"]
+        else:
+            self._last_price = float(prices["Close"].iloc[-1])
+            if _np.isnan(self._last_price):
+                md = self._get_exchange_metadata()
+                if "regularMarketPrice" in md:
+                    self._last_price = md["regularMarketPrice"]
+        return self._last_price
+
+    @property
+    def previous_close(self):
+        if self._prev_close is not None:
+            return self._prev_close
+        prices = self._get_1wk_1h_prepost_prices()
+        fail = False
+        if prices.empty:
+            fail = True
+        else:
+            prices = prices[["Close"]].groupby(prices.index.date).last()
+            if prices.shape[0] < 2:
+                # Very few symbols have previousClose despite no 
+                # no trading data e.g. 'QCSTIX'.
+                fail = True
+            else:
+                self._prev_close = float(prices["Close"].iloc[-2])
+        if fail:
+            # Fallback to original info[] if available.
+            self._tkr.info  # trigger fetch
+            k = "previousClose"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._prev_close = self._tkr._quote._retired_info[k]
+        return self._prev_close
+
+    @property
+    def regular_market_previous_close(self):
+        if self._reg_prev_close is not None:
+            return self._reg_prev_close
+        prices = self._get_1y_prices()
+        if prices.shape[0] == 1:
+            # Tiny % of tickers don't return daily history before last trading day, 
+            # so backup option is hourly history:
+            prices = self._get_1wk_1h_reg_prices()
+            prices = prices[["Close"]].groupby(prices.index.date).last()
+        if prices.shape[0] < 2:
+            # Very few symbols have regularMarketPreviousClose despite no 
+            # no trading data. E.g. 'QCSTIX'.
+            # So fallback to original info[] if available.
+            self._tkr.info  # trigger fetch
+            k = "regularMarketPreviousClose"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._reg_prev_close = self._tkr._quote._retired_info[k]
+        else:
+            self._reg_prev_close = float(prices["Close"].iloc[-2])
+        return self._reg_prev_close
+
+    @property
+    def open(self):
+        if self._open is not None:
+            return self._open
+        prices = self._get_1y_prices()
+        if prices.empty:
+            self._open = None
+        else:
+            self._open = float(prices["Open"].iloc[-1])
+            if _np.isnan(self._open):
+                self._open = None
+        return self._open
+
+    @property
+    def day_high(self):
+        if self._day_high is not None:
+            return self._day_high
+        prices = self._get_1y_prices()
+        if prices.empty:
+            self._day_high = None
+        else:
+            self._day_high = float(prices["High"].iloc[-1])
+            if _np.isnan(self._day_high):
+                self._day_high = None
+        return self._day_high
+
+    @property
+    def day_low(self):
+        if self._day_low is not None:
+            return self._day_low
+        prices = self._get_1y_prices()
+        if prices.empty:
+            self._day_low = None
+        else:
+            self._day_low = float(prices["Low"].iloc[-1])
+            if _np.isnan(self._day_low):
+                self._day_low = None
+        return self._day_low
+
+    @property
+    def last_volume(self):
+        if self._last_volume is not None:
+            return self._last_volume
+        prices = self._get_1y_prices()
+        self._last_volume = None if prices.empty else int(prices["Volume"].iloc[-1])
+        return self._last_volume
+
+    @property
+    def fifty_day_average(self):
+        if self._50d_day_average is not None:
+            return self._50d_day_average
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._50d_day_average = None
+        else:
+            n = prices.shape[0]
+            a = n-50
+            b = n
+            if a < 0:
+                a = 0
+            self._50d_day_average = float(prices["Close"].iloc[a:b].mean())
+
+        return self._50d_day_average
+
+    @property
+    def two_hundred_day_average(self):
+        if self._200d_day_average is not None:
+            return self._200d_day_average
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._200d_day_average = None
+        else:
+            n = prices.shape[0]
+            a = n-200
+            b = n
+            if a < 0:
+                a = 0
+
+            self._200d_day_average = float(prices["Close"].iloc[a:b].mean())
+
+        return self._200d_day_average
+
+    @property
+    def ten_day_average_volume(self):
+        if self._10d_avg_vol is not None:
+            return self._10d_avg_vol
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._10d_avg_vol = None
+        else:
+            n = prices.shape[0]
+            a = n-10
+            b = n
+            if a < 0:
+                a = 0
+            self._10d_avg_vol = int(prices["Volume"].iloc[a:b].mean())
+
+        return self._10d_avg_vol
+
+    @property
+    def three_month_average_volume(self):
+        if self._3mo_avg_vol is not None:
+            return self._3mo_avg_vol
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            self._3mo_avg_vol = None
+        else:
+            dt1 = prices.index[-1]
+            dt0 = dt1 - utils._interval_to_timedelta("3mo") + utils._interval_to_timedelta("1d")
+            self._3mo_avg_vol = int(prices.loc[dt0:dt1, "Volume"].mean())
+
+        return self._3mo_avg_vol
+
+    @property
+    def year_high(self):
+        if self._year_high is not None:
+            return self._year_high
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            prices = self._get_1y_prices(fullDaysOnly=False)
+        self._year_high = float(prices["High"].max())
+        return self._year_high
+
+    @property
+    def year_low(self):
+        if self._year_low is not None:
+            return self._year_low
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.empty:
+            prices = self._get_1y_prices(fullDaysOnly=False)
+        self._year_low = float(prices["Low"].min())
+        return self._year_low
+
+    @property
+    def year_change(self):
+        if self._year_change is not None:
+            return self._year_change
+
+        prices = self._get_1y_prices(fullDaysOnly=True)
+        if prices.shape[0] >= 2:
+            self._year_change = (prices["Close"].iloc[-1] - prices["Close"].iloc[0]) / prices["Close"].iloc[0]
+            self._year_change = float(self._year_change)
+        return self._year_change
+
+    @property
+    def market_cap(self):
+        if self._mcap is not None:
+            return self._mcap
+
+        try:
+            shares = self.shares
+        except Exception as e:
+            if "Cannot retrieve share count" in str(e):
+                shares = None
+            else:
+                raise
+
+        if shares is None:
+            # Very few symbols have marketCap despite no share count.
+            # E.g. 'BTC-USD'
+            # So fallback to original info[] if available.
+            self._tkr.info
+            k = "marketCap"
+            if self._tkr._quote._retired_info is not None and k in self._tkr._quote._retired_info:
+                self._mcap = self._tkr._quote._retired_info[k]
+        else:
+            self._mcap = float(shares * self.last_price)
+        return self._mcap
+
 
 class Quote:
 
@@ -87,13 +545,16 @@ class Quote:
         self._calendar = None
 
         self._already_scraped = False
-        self._already_scraped_complementary = False
+        self._already_fetched = False
+        self._already_fetched_complementary = False
 
     @property
     def info(self) -> dict:
         if self._info is None:
-            self._scrape(self.proxy)
-            self._scrape_complementary(self.proxy)
+            # self._scrape(self.proxy)  # decrypt broken
+            self._fetch(self.proxy)
+
+            self._fetch_complementary(self.proxy)
 
         return self._info
 
@@ -236,12 +697,56 @@ class Quote:
         except Exception:
             pass
 
-    def _scrape_complementary(self, proxy):
-        if self._already_scraped_complementary:
+    def _fetch(self, proxy):
+        if self._already_fetched:
             return
-        self._already_scraped_complementary = True
+        self._already_fetched = True
+        modules = ['summaryProfile', 'financialData', 'quoteType',
+                     'defaultKeyStatistics', 'assetProfile', 'summaryDetail']
+        result = self._data.get_raw_json(
+            _BASIC_URL_ + f"/{self._data.ticker}", params={"modules": ",".join(modules), "ssl": "true"}, proxy=proxy
+        )
+        result["quoteSummary"]["result"][0]["symbol"] = self._data.ticker
+        query1_info = next(
+            (info for info in result.get("quoteSummary", {}).get("result", []) if info["symbol"] == self._data.ticker),
+            None,
+        )
+        # Most keys that appear in multiple dicts have same value. Except 'maxAge' because
+        # Yahoo not consistent with days vs seconds. Fix it here:
+        for k in query1_info:
+            if "maxAge" in query1_info[k] and query1_info[k]["maxAge"] == 1:
+                query1_info[k]["maxAge"] = 86400
+        query1_info = {
+            k1: v1 
+            for k, v in query1_info.items() 
+            if isinstance(v, dict) 
+            for k1, v1 in v.items() 
+            if v1
+        }
+        # recursively format but only because of 'companyOfficers'
+        def _format(k, v):
+            if isinstance(v, dict) and "raw" in v and "fmt" in v:
+                v2 = v["fmt"] if k in {"regularMarketTime", "postMarketTime"} else v["raw"]
+            elif isinstance(v, list):
+                v2 = [_format(None, x) for x in v]
+            elif isinstance(v, dict):
+                v2 = {k:_format(k, x) for k, x in v.items()}
+            elif isinstance(v, str):
+                v2 = v.replace("\xa0", " ")
+            else:
+                v2 = v
+            return v2
+        for k, v in query1_info.items():
+            query1_info[k] = _format(k, v)
+        self._info = query1_info
 
-        self._scrape(proxy)
+    def _fetch_complementary(self, proxy):
+        if self._already_fetched_complementary:
+            return
+        self._already_fetched_complementary = True
+
+        # self._scrape(proxy)  # decrypt broken
+        self._fetch(proxy)
         if self._info is None:
             return
 
@@ -283,11 +788,14 @@ class Quote:
 
             json_str = self._data.cache_get(url=url, proxy=proxy).text
             json_data = json.loads(json_str)
-            key_stats = json_data["timeseries"]["result"][0]
-            if k not in key_stats:
-                # Yahoo website prints N/A, indicates Yahoo lacks necessary data to calculate
+            try:
+                key_stats = json_data["timeseries"]["result"][0]
+                if k not in key_stats:
+                    # Yahoo website prints N/A, indicates Yahoo lacks necessary data to calculate
+                    v = None
+                else:
+                    # Select most recent (last) raw value in list:
+                    v = key_stats[k][-1]["reportedValue"]["raw"]
+            except Exception:
                 v = None
-            else:
-                # Select most recent (last) raw value in list:
-                v = key_stats[k][-1]["reportedValue"]["raw"]
             self._info[k] = v

yfinance/scrapers/yahoo-keys.txt

@@ -3,3 +3,6 @@ ad4d90b3c9f2e1d156ef98eadfa0ff93e4042f6960e54aa2a13f06f528e6b50ba4265a26a1fd5b9c
 e9a8ab8e5620b712ebc2fb4f33d5c8b9c80c0d07e8c371911c785cf674789f1747d76a909510158a7b7419e86857f2d7abbd777813ff64840e4cbc514d12bcae
 6ae2523aeafa283dad746556540145bf603f44edbf37ad404d3766a8420bb5eb1d3738f52a227b88283cca9cae44060d5f0bba84b6a495082589f5fe7acbdc9e
 3365117c2a368ffa5df7313a4a84988f73926a86358e8eea9497c5ff799ce27d104b68e5f2fbffa6f8f92c1fef41765a7066fa6bcf050810a9c4c7872fd3ebf0
+15d8f57919857d5a5358d2082c7ef0f1129cfacd2a6480333dcfb954b7bb67d820abefebfdb0eaa6ef18a1c57f617b67d7e7b0ec040403b889630ae5db5a4dbb
+db9630d707a7d0953ac795cd8db1ca9ca6c9d8239197cdfda24b4e0ec9c37eaec4db82dab68b8f606ab7b5b4af3e65dab50606f8cf508269ec927e6ee605fb78
+3c895fb5ddcc37d20d3073ed74ee3efad59bcb147c8e80fd279f83701b74b092d503dcd399604c6d8be8f3013429d3c2c76ed5b31b80c9df92d5eab6d3339fce

yfinance/utils.py

@@ -35,6 +35,7 @@ import os as _os
 import appdirs as _ad
 import sqlite3 as _sqlite3
 import atexit as _atexit
+from functools import lru_cache
 
 from threading import Lock
 
@@ -61,6 +62,13 @@ def attributes(obj):
         if name[0] != '_' and name not in disallowed_names and hasattr(obj, name)}
 
 
+@lru_cache(maxsize=20)
+def print_once(msg):
+    # 'warnings' module suppression of repeat messages does not work. 
+    # This function replicates correct behaviour
+    print(msg)
+
+
 def is_isin(string):
     return bool(_re.match("^([A-Z]{2})([A-Z0-9]{9})([0-9]{1})$", string))
 
@@ -552,7 +560,7 @@ def safe_merge_dfs(df_main, df_sub, interval):
 
         df["_NewIndex"] = new_index
         # Duplicates present within periods but can aggregate
-        if data_col_name == "Dividends":
+        if data_col_name in ["Dividends", "Capital Gains"]:
             # Add
             df = df.groupby("_NewIndex").sum()
             df.index.name = None
@@ -699,7 +707,7 @@ def format_history_metadata(md):
     tz = md["exchangeTimezoneName"]
 
     for k in ["firstTradeDate", "regularMarketTime"]:
-        if k in md:
+        if k in md and md[k] is not None:
             md[k] = _pd.to_datetime(md[k], unit='s', utc=True).tz_convert(tz)
 
     if "currentTradingPeriod" in md:
@@ -912,6 +920,8 @@ class _TzCache:
             df = _pd.read_csv(old_cache_file_path, index_col="Ticker")
         except _pd.errors.EmptyDataError:
             _os.remove(old_cache_file_path)
+        except TypeError:
+            _os.remove(old_cache_file_path)
         else:
             self.tz_db.bulk_set(df.to_dict()['Tz'])
             _os.remove(old_cache_file_path)

yfinance/version.py

@@ -1 +1 @@
-version = "0.2.10b2"
+version = "0.2.18"