- Add a new class Screener, Query, and EquityQuery
- Refactor YfData.get() to enable YfData.post() to all of get's implementations
- Add test for the Screener
- Add new set and map to const
Screener can be used to filter yahoo finance. This can be used to get the top gainers of the day, and more customized queries that a user may want to automate.
Sector and Industry class to query the data available on: https://finance.yahoo.com/sectors/:
- sector / industry overview
- sector / industry top companies
- sector / industry research reports
- sector top ETFs
- sector top Mutual Funds
- industries within sector
- industry's sector
- industry's top performing companies
- industry's top growth companies
- adding to scraper a new class FundsData.
- Ticker can now reference the data via ticker.funds_data when appropriate.
- Add unit tests in test_ticker.py for the funds data
The new dividend repair logic looks for too big/small dividends/adjusts.
Because it analyses prices, it needs prices to be consistent - this means
for some exchanges, prices are converted to major currency e.g. pence -> £.
These exchanges are: .L, .TA, .JO
Also, multiday intervals are created by resampling adjusted 1d intervals.
Related change:
- _fix_prices_sudden_change() now fixes dividends with prices
Other repair fixes:
- add basic repair function to fix_Yahoo_returning_live_separate() to handle simple 100x price errors within same week/month
- fix _fix_prices_sudden_change() false positive on real massive price drop (clue = massive volume spike)
- price reconstruct fixes:
- calibration: stop NaNs breaking, and detect currency unit mixup
- fix a Pandas warning
Tidy messages sent to logging.INFO
Use logging's 'extra' argument to make YF log formatting programmatic.
Reclassify some YF price-repair log messages, so that repairs (or repair fails) are level INFO.
Original logic for repairing missing split adjustment only checked latest split.
Improved logic checks ALL splits in data, because any can be missing.
Then related changes to 'sudden change detection':
- use prices median not mean, reduce sensitivity to noise.
- handle Kuwait Dinar, which sub-divides into 1000x not 100x.
Session switch logic was not recalculating 'self._session_is_caching'.
Also removed message 'help stress-test cookie & crumb & requests_cache', clearly works now.
Python 3.12 deprecates datetime.datetime.utcnow().
Instead of switching to datetime.datetime.now(datetime.UTC), which won't work in Python 3.11,
just switch to Pandas.utcnow().
Error in supplying timescale values resulted in misleading ValueError -
```
ValueError: Illegal argument: timescale must be one of: ['income', 'balance-sheet', 'cash-flow']
```
'peewee.DateTimeField' is not ISO-compliant. If user enforces strict ISO-compliance,
then translation between DateTimeField and sqlite breaks. Fix is to manually
implement translation.
Add cookie & crumb to requests. Involves several changes:
- fetch cookie & crumb, obviously.
- two different cookie strategies - one seems to work better in USA, other better outside.
- yfinance auto-detects if one strategy fails, and switches to other strategy.
- cookie is stored in persistent cache folder, alongside timezones. Refetched after 24 hours.
To have this work well with multithreading (yfinance.download()) requires more changes:
- all threads share the same cookie, therefore the same session object. Requires thread-safety ...
- converted data class to a singleton with "SingletonMeta":
- the first init() call initialises data.
- but successive calls update its session object - naughty but necessary.
- thread locks to avoid deadlocks and race conditions.
The initial singleton design pattern for database access meant that lazy-loading was broken,
due to structure of '_KV' class. So errors were blocking import.
Fix = use 'peewee' proxy database and initialise when needed.
If Yahoo returns intraday price data with dividend or stock-split event in future, then this broke the merge.
Fix is to discard out-of-range events.
Assumes that if user requesting intraday then they aren't interested in events.
Price repair fixes and improvement
Fixes:
- fix reconstruction mis-calibration with tiny DataFrames
- fix detecting last-active-trading-interval when NaNs in DataFrame
- redesign mapping 100x signals to ranges:
- no change for signals before last-active-trading-interval
- but for signals after last-active-trading-interval, process in reverse order
Improvements:
- increase max reconstruction depth from 1 to 2. E.g. now 1wk can be repaired with 1h (1wk->1d->1h)
Several improvements to price repair
Repair 100x and split errors:
- Handle stocks that recently suspended - use latest ACTIVE trading as baseline
- Improve error identification:
- Restrict repair to no older than 1 year before oldest split
- To reduce false positives when checking for multiday split errors,
only analyse 'Open' and 'Close' and use average change instead of nearest-to-1
- For weekly intervals reduce threshold to 3x standard deviation (5x was too high),
and for monthly increase to 6x
- For multiday intervas, if errors only detected in 1 column then assume false positive => ignore
Repair missing div-adjust:
- Fix repair of multiday intervals containing dividend
Price reconstruction:
- Move to after repairing 100x and split errors, so calibration works properly
- Fix maximum depth and reduce to 1
- Restrict calibration to 'Open' and 'Close', because 'Low' and 'High' can differ significantly between e.g. 1d and day-of-1h
Miscellaneous:
- Deprecate repair='silent', the logging module handles this
- Improve tests for 100x and split errors
- New test for 'repair missing div adjust'
timestamp of 1900 is older than 100 years,
so yahoo responds with error:
GDEVW: 1d data not available for startTime=-2208994789 and
endTime=1687780922. Only 100 years worth of day granularity data are
allowed to be fetched per request.
this should fix it,
something similar was proposed here:
https://github.com/ranaroussi/yfinance/pull/648
# Please enter the commit message for
your changes. Lines starting
Improve split-repair of multi-day intervals. Because split error can occur within a multi-day interval, e.g. mid-way through week, need to repair each OHLC column separately
Increase robustness of repair 'Adj Close'
Limit price-repair recursion depth to 2
Various important changes to yfinance logging:
Remove handler from YF logger as breaks standard logging practice.
Improve 'download' log message grouping
Move custom DEBUG log formatting into 'yf.enable_debug_mode()', which if called adds the handler. The custom DEBUG log formatting is:
- 'MultiLineFormatter' improves appearance of multi-line messages in any logging mode. Adds leading indent to all lines below first, to align with the first line which is indented by %levelname%.
- Whitespace padding is added to end of %levelname% string up to 8 characters. This resolves different level strings e.g. INFO & DEBUG creating different indents. But this only automatically applies to first line of message - for multi-line messages, 'MultiLineFormatter' extracts amount of padding and applies to all other lines.
- Add leading indent proportional to function depth, because DEBUG generates a lot of messages particularly when repairing price data - same function can be called recursively. Implemented in 'IndentLoggerAdapter', a simple wrapper around the logger object.
- 'log_indent_decorator' inserts 'Entering/Exiting %function%' debug messages, helps organise.
Main change is fixing price-repair of 1d 'Adj Close'. 1d repair uses 1h data, but 1h is never div-adjusted. For correct 1d 'Adj Close', extract div-adjustment from the good 1d data, and calculate it for any bad 1d data. A new unit test ensures correctness.
Other changes:
- bug fix in split-repair logic to handle price=0
- improve unit test coverage on price dividend
- add 1wk interval to split-repair unit test
Stumbled upon another type of 100x price error - Yahoo may switch a symbol from e.g. cents -> $ on some recent date, so recent prices are 100x different. The new split-repair is perfect for this - set change to 100 and ignore 'Volume'.
An out-of-range dividend was breaking merge with 1mo-prices, so fixed that. Also replaced the mega-loop with Numpy, much clearer now. Improved its tests.
Recent release action generated deprecated error: "Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/setup-python@v2."
So simply increasing versions to match latest GitHub usage docs, hopefully works.
New logging in `download` stores the tracebacks, but the logic was faulty, this fixes that.
Also improves error handling in `download`.
Unit tests should have detected this so improved them:
- add/improve `download` tests
- disable tests that require Yahoo decryption (because is broken)
- fix logging-related errors
- improve session use
Improve logging messages related to price data fetches:
- fix 'debug is deprecated' msg
- append user args to 'may be delisted' msg - interval & dates/period
- improve formatting of 'cannot reconstruct' msg
- hide errors in 'history()' while accessing 'fast_info[]'
format_history_metadata() is expensive. Improvements:
- only perform full formatting if user requests metadata
- when pruning prepost data, only format 'tradingPeriods' entry of metadata
Other small optimisations to several internal prices processing methods.
Speedups:
dat.history(period='1wk', interval='1h', prepost=True) # 2x
dat.history(period='1mo', interval='1h', prepost=True) # 1.46x
dat.history(period='1wk', interval='1h') # 1.15x
dat.history(period='1mo', interval='1h') # 1.13x
dat.history(period='1y', interval='1d') # 1.36x
dat.history(period='5y', interval='1d') # 1.13x
* Changed base url to "https://query2.finance.yahoo.com/v10/finance/quoteSummary"
* instead of just getting the quote, we now get
* ```
items = ['summaryProfile', 'financialData', 'quoteType',
'defaultKeyStatistics', 'assetProfile', 'summaryDetail']
```
which is the same as in the scrape function
If you want help, you got to read this first, follow the instructions.
### Are you up-to-date?
Upgrade to the latest version and confirm the issue/bug is still there.
`$ pip install yfinance --upgrade --no-cache-dir`
Confirm by running:
`import yfinance as yf ; print(yf.__version__)`
and comparing against [PIP](https://pypi.org/project/yfinance/#history).
### Does Yahoo actually have the data?
Are you spelling ticker *exactly* same as Yahoo?
Then visit `finance.yahoo.com` and confirm they have the data you want. Maybe your ticker was delisted, or your expectations of `yfinance` are wrong.
### Are you spamming Yahoo?
Yahoo Finance free service has rate-limiting depending on request type - roughly 60/minute for prices, 10/minute for info. Once limit hit, Yahoo can delay, block, or return bad data. Not a `yfinance` bug.
### Still think it's a bug?
Delete this default message (all of it) and submit your bug report here, providing the following as best you can:
- Simple code that reproduces your problem, that we can copy-paste-run
- Exception message with full traceback, or proof `yfinance` returning bad data
# !!! IMPORTANT !!! FOLLOW THESE INSTRUCTIONS CAREFULLY !!!
### Are you up-to-date?
Upgrade to the latest version: `$ pip install yfinance --upgrade --no-cache-dir`
Confirm latest version by running: `import yfinance as yf ; print(yf.__version__)` and comparing against [PyPI](https://pypi.org/project/yfinance/#history).
### Does Yahoo actually have the data?
Are you spelling symbol *exactly* same as Yahoo?
Then visit `finance.yahoo.com` and confirm they have the data you want. Maybe your symbol was delisted, or your expectations of `yfinance` are wrong.
### Are you spamming Yahoo?
Yahoo Finance free service has rate-limiting https://github.com/ranaroussi/yfinance/discussions/1513. Once limit hit, Yahoo can delay, block, or return bad data -> not a `yfinance` bug.
### Does issue already exist?
Use the search tool. Don't duplicate existing issues.
- type:markdown
attributes:
value:|
---
## Still think it's a bug?
Provide the following as best you can:
- type:textarea
id:summary
attributes:
label:"Describe bug"
validations:
required:true
- type:textarea
id:code
attributes:
label:"Simple code that reproduces your problem"
description:"Provide a snippet of code that we can copy-paste-run. Wrap code in Python Markdown code blocks for proper formatting (```` ```python ... ``` ````)."
validations:
required:true
- type:textarea
id:debug-log
attributes:
label:"Debug log"
description:"Run code with debug logging enabled - `yf.enable_debug_mode()` - and post the full output. Context: https://github.com/ranaroussi/yfinance/tree/main#logging"
validations:
required:true
- type:textarea
id:bad-data-proof
attributes:
label:"Bad data proof"
description:"If `yfinance` returning bad data, show proof here. Best is screenshot of finance.yahoo.com"
<a target="new" href="https://github.com/ranaroussi/yfinance"><img border=0 src="https://img.shields.io/github/stars/ranaroussi/yfinance.svg?style=social&label=Star&maxAge=60" alt="Star this repo"></a>
<a target="new" href="https://twitter.com/aroussi"><img border=0 src="https://img.shields.io/twitter/follow/aroussi.svg?style=social&label=Follow&maxAge=60" alt="Follow me on twitter"></a>
@@ -36,234 +35,16 @@ Yahoo! finance API is intended for personal use only.**
**yfinance** offers a threaded and Pythonic way to download market data from [Yahoo!Ⓡ finance](https://finance.yahoo.com).
→ Check out this [Blog post](https://aroussi.com/#post/python-yahoo-finance) for a detailed tutorial with code examples.
## Main Features
-`Ticker` module: Class for accessing single ticker data.
-`Tickers` module: Class for handling multiple tickers.
-`download` Efficiently download market data for multiple tickers.
-`Sector` and `Industry` modules : Classes for accessing sector and industry information.
- Market Screening: `EquityQuery` and `Screener` to build query and screen the market.
Since December 2022 Yahoo has been encrypting the web data that `yfinance` scrapes for non-market data. Fortunately the decryption keys are available, although Yahoo moved/changed them several times hence `yfinance` breaking several times. `yfinance` is now better prepared for any future changes by Yahoo.
Why is Yahoo doing this? We don't know. Is it to stop scrapers? Maybe, so we've implemented changes to reduce load on Yahoo. In December we rolled out version 0.2 with optimised scraping. Then in 0.2.6 introduced `Ticker.fast_info`, providing much faster access to some `info` elements wherever possible e.g. price stats and forcing users to switch (sorry but we think necessary). `info` will continue to exist for as long as there are elements without a fast alternative.
## Quick Start
### The Ticker module
The `Ticker` module, which allows you to access ticker data in a more Pythonic way:
```python
importyfinanceasyf
msft=yf.Ticker("MSFT")
# get all stock info (slow)
msft.info
# fast access to subset of stock info (opportunistic)
msft.fast_info
# get historical market data
hist=msft.history(period="1mo")
# show meta information about the history (requires history() to be called first)
msft.history_metadata
# show actions (dividends, splits, capital gains)
msft.actions
msft.dividends
msft.splits
msft.capital_gains# only for mutual funds & etfs
# show share count
# - yearly summary:
msft.shares
# - accurate time-series count:
msft.get_shares_full(start="2022-01-01",end=None)
# show financials:
# - income statement
msft.income_stmt
msft.quarterly_income_stmt
# - balance sheet
msft.balance_sheet
msft.quarterly_balance_sheet
# - cash flow statement
msft.cashflow
msft.quarterly_cashflow
# see `Ticker.get_income_stmt()` for more options
# show holders
msft.major_holders
msft.institutional_holders
msft.mutualfund_holders
# show earnings
msft.earnings
msft.quarterly_earnings
# show sustainability
msft.sustainability
# show analysts recommendations
msft.recommendations
msft.recommendations_summary
# show analysts other work
msft.analyst_price_target
msft.revenue_forecasts
msft.earnings_forecasts
msft.earnings_trend
# show next event (earnings, etc)
msft.calendar
# Show future and historic earnings dates, returns at most next 4 quarters and last 8 quarters by default.
# Note: If more are needed use msft.get_earnings_dates(limit=XX) with increased limit argument.
msft.earnings_dates
# show ISIN code - *experimental*
# ISIN = International Securities Identification Number
msft.isin
# show options expirations
msft.options
# show news
msft.news
# get option chain for specific expiration
opt=msft.option_chain('YYYY-MM-DD')
# data available via: opt.calls, opt.puts
```
If you want to use a proxy server for downloading data, use:
To install with optional dependencies, replace `optional` with: `nospam` for [caching-requests](#smarter-scraping), `repair` for [price repair](https://github.com/ranaroussi/yfinance/wiki/Price-repair), or `nospam,repair` for both:
3. Create a new branch for your feature or bug fix:
..code-block::bash
git checkout -b feature-branch-name
4. Make your changes, commit them, and push your branch to GitHub. To keep the commit history and `network graph <https://github.com/ranaroussi/yfinance/network>`_ compact:
-**dev**: New features and some bug fixes are merged here. This branch allows collective testing, conflict resolution, and further stabilization before merging into the stable branch.
-**main**: Stable branch where PIP releases are created.
By default, branches target **main**, but most contributions should target **dev**.
**Exceptions**:
Direct merges to **main** are allowed if:
-`yfinance` is massively broken
- Part of `yfinance` is broken, and the fix is simple and isolated
Unit Tests
----------
Tests are written using Python’s `unittest` module. Here are some ways to run tests:
* yfinance documentation is written in reStructuredText (rst) and built using Sphinx.
* The documentation file is in `doc/source/..`.
* Most of the notes under API References read from class and methods docstrings. These documentations, found in `doc/source/reference/api` is autogenerated by Sphinx and not included in git.
Building documentation locally
-------------------------------
To build the documentation locally, follow these steps:
1.**Install Required Dependencies**:
* Make sure `Sphinx` and any other dependencies are installed. If a `requirements.txt` file is available, you can install dependencies by running:
..code-block::console
pip install -r requirements.txt
2.**Build with Sphinx**:
* After dependencies are installed, use the sphinx-build command to generate HTML documentation.
* Go to `doc/` directory Run:
..code-block::console
make clean && make html
3.**View Documentation Locally**:
* Open `doc/build/html/index.html` in the browser to view the generated documentation.
Building documentation on main
-------------------------------
The documentation updates are built on merge to `main` branch. This is done via GitHub Actions workflow based on `/yfinance/.github/workflows/deploy_doc.yml`.
1. Reivew the changes locally and push to `dev`.
2. When `dev` gets merged to `main`, GitHub Actions workflow is automated to build documentation.
To install with optional dependencies, replace `optional` with: `nospam` for `caching-requests <https://github.com/ranaroussi/yfinance?tab=readme-ov-file#smarter-scraping>`_, `repair` for `price repair <https://github.com/ranaroussi/yfinance/wiki/Price-repair>`_, or `nospam`, `repair` for both:
..code-block::bash
$ pip install "yfinance[optional]"
For required dependencies, check out the `requirements file <./requirements.txt>`_, and for all dependencies, see the `setup.py file <./setup.py#L62>`_.
yfinance is distributed under the Apache Software License. See the `LICENSE.txt <../../../../LICENSE.txt>`_ file for details.
Again, yfinance is **not** affiliated, endorsed, or vetted by Yahoo, Inc. It's an open-source tool that uses Yahoo's publicly available APIs, and is intended for research and educational purposes.
The `yfinance` package provides easy access to Yahoo! Finance's API to retrieve market data. It includes classes and functions for downloading historical market data, accessing ticker information, managing cache, and more.
Public API
==========
The following are the publicly available classes, and functions exposed by the `yfinance` package:
-:attr:`Ticker <yfinance.Ticker>`: Class for accessing single ticker data.
-:attr:`Tickers <yfinance.Tickers>`: Class for handling multiple tickers.
-:attr:`Sector <yfinance.Sector>`: Domain class for accessing sector information.
-:attr:`Industry <yfinance.Industry>`: Domain class for accessing industry information.
-:attr:`download <yfinance.download>`: Function to download market data for multiple tickers.
-:attr:`EquityQuery <yfinance.EquityQuery>`: Class to build equity market query.
-:attr:`Screener <yfinance.Screener>`: Class to screen the market using defined query.
-:attr:`enable_debug_mode <yfinance.enable_debug_mode>`: Function to enable debug mode for logging.
-:attr:`set_tz_cache_location <yfinance.set_tz_cache_location>`: Function to set the timezone cache location.
3. Create a new branch for your feature or bug fix:
.. code-block:: bash
git checkout -b feature-branch-name
4. Make your changes, commit them, and push your branch to GitHub. To keep the commit history and `network graph <https://github.com/ranaroussi/yfinance/network>`_ compact:
- **dev**: New features and some bug fixes are merged here. This branch allows collective testing, conflict resolution, and further stabilization before merging into the stable branch.
- **main**: Stable branch where PIP releases are created.
By default, branches target **main**, but most contributions should target **dev**.
**Exceptions**:
Direct merges to **main** are allowed if:
- `yfinance` is massively broken
- Part of `yfinance` is broken, and the fix is simple and isolated
Unit Tests
----------
Tests are written using Python’s `unittest` module. Here are some ways to run tests:
* yfinance documentation is written in reStructuredText (rst) and built using Sphinx.
* The documentation file is in `doc/source/..`.
* Most of the notes under API References read from class and methods docstrings. These documentations, found in `doc/source/reference/api` is autogenerated by Sphinx and not included in git.
Building documentation locally
-------------------------------
To build the documentation locally, follow these steps:
1. **Install Required Dependencies**:
* Make sure `Sphinx` and any other dependencies are installed. If a `requirements.txt` file is available, you can install dependencies by running:
.. code-block:: console
pip install -r requirements.txt
2. **Build with Sphinx**:
* After dependencies are installed, use the sphinx-build command to generate HTML documentation.
* Go to `doc/` directory Run:
.. code-block:: console
make clean && make html
3. **View Documentation Locally**:
* Open `doc/build/html/index.html` in the browser to view the generated documentation.
Building documentation on main
-------------------------------
The documentation updates are built on merge to `main` branch. This is done via GitHub Actions workflow based on `/yfinance/.github/workflows/deploy_doc.yml`.
1. Reivew the changes locally and push to `dev`.
2. When `dev` gets merged to `main`, GitHub Actions workflow is automated to build documentation.
To install with optional dependencies, replace `optional` with: `nospam` for `caching-requests <https://github.com/ranaroussi/yfinance?tab=readme-ov-file#smarter-scraping>`_, `repair` for `price repair <https://github.com/ranaroussi/yfinance/wiki/Price-repair>`_, or `nospam`, `repair` for both:
.. code-block:: bash
$ pip install "yfinance[optional]"
For required dependencies, check out the `requirements file <./requirements.txt>`_, and for all dependencies, see the `setup.py file <./setup.py#L62>`_.
yfinance is distributed under the Apache Software License. See the `LICENSE.txt <../../../../LICENSE.txt>`_ file for details.
Again, yfinance is **not** affiliated, endorsed, or vetted by Yahoo, Inc. It's an open-source tool that uses Yahoo's publicly available APIs, and is intended for research and educational purposes.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
