Python 係咪用Selenium 係Web Scraping 唔到Bloomberg?

wood_0220

如題，我宜家都係想去Bloomberg 去拎恒生指數的PE 數值，如下圖：



但有兩個問題：
1）Bloomberg detect 我係bot, 看下圖：


2) Bloomberg 個網成日load 緊野，由於網頁一直load, python 會等個website load 晒先會執行下一行code，變相都係拎唔到數值。


import  ws_functions.config                 as ws_config
import  ws_functions.cust_functions         as ws_functions
import  time
import  sqlite3
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys


ws_functions.print_runtime()

bloomberg_frontpage_url = 'https://www.bloomberg.com/asia'

options = webdriver.ChromeOptions()
options.binary_location = "/Applications/Google Chrome.app"
# chrome_driver_binary = '/Users/seansiao/Documents/workspace/HK Property/Valuation/chromedriver'
chrome_options = Options()
#chrome_options.add_argument("--headless")

chrome_options.add_argument("window-size=1920,1080")
browser=webdriver.Chrome(chrome_options=chrome_options, executable_path=r'/Users/ws_fingear/Documents/workspace/chromedriver')
chrome_options = Options()
browser.get(bloomberg_frontpage_url)


browser.implicitly_wait(10)

try:
    time.sleep(6)
    browser.find_element_by_xpath('//*[@id="nav-bar-search-button"]').click() #Click "Search"
    time.sleep(5)
    browser.find_element_by_xpath('//*[@id="navi-search-input"]').send_keys('HSI') #Send 'HSI' to the searchbar
    time.sleep(5)
    browser.find_element_by_partial_link_text('HSI:IND').click()
    time.sleep(5)
    time.sleep(5)
    HSI_PE = browser.find_element_by_xpath('//[@id="root"]/div/div/section[3]/div[3]/div[1]/div/div[1]/div[1]/div/span[2]').text


可唔可以有人救救我？感謝大家

pchai

用正統既API Call
free既當然有限制

https://www.gugudata.com/api/details/stockhkrealtime

https://polygon.io/pricing

javacomhk

用Javascript 都可以.

javacomhk

用得 Bloomberg data 一定唔係港股咁簡單啦。 Bloomberg 的 data license 好貴架，遂個 market 同你計。

pchai

1. import requests
   
2. HSI_url = 'https://www.bloomberg.com/markets2/api/datastrip/HSI%3AIND%2CINDU%3AIND%2CSPX%3AIND?locale=en&customTickerList=true'
   
3. headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"}
   
4. resp = requests.get(HSI_url, headers=headers)
   
5. data = resp.json()
   
6. print(data[0]['priceEarningsRatio'])

複製代碼

wood_0220

pchai 發表於 2021-7-15 15:45

帥兄呢個方法，連API 都好似唔駛用...

javacomhk

If you use website that has api provided, it is fast but may requires charges from the website.

If you use selenium, you are dealing with login credential and sometimes login captcha and javascript dynamic page.

If the destination page has url address, you can simply grab these static page content and put it in BeautifulSoup or json and based on the css selector print the results as described here. https://towardsdatascience.com/web-scraping-basics-82f8b5acd45c