標題: Python 係咪用Selenium 係Web Scraping 唔到Bloomberg? [打印本頁]

---

作者: wood_0220    時間: 2021-7-13 23:14     標題: Python 係咪用Selenium 係Web Scraping 唔到Bloomberg?

如題，我宜家都係想去Bloomberg 去拎恒生指數的PE 數值，如下圖：

[attach]2272672[/attach]

但有兩個問題：
1）Bloomberg detect 我係bot, 看下圖：
[attach]2272673[/attach]

2) Bloomberg 個網成日load 緊野，由於網頁一直load, python 會等個website load 晒先會執行下一行code，變相都係拎唔到數值。


import  ws_functions.config                 as ws_config
import  ws_functions.cust_functions         as ws_functions
import  time
import  sqlite3
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys


ws_functions.print_runtime()

bloomberg_frontpage_url = 'https://www.bloomberg.com/asia'

options = webdriver.ChromeOptions()
options.binary_location = "/Applications/Google Chrome.app"
# chrome_driver_binary = '/Users/seansiao/Documents/workspace/HK Property/Valuation/chromedriver'
chrome_options = Options()
#chrome_options.add_argument("--headless")

chrome_options.add_argument("window-size=1920,1080")
browser=webdriver.Chrome(chrome_options=chrome_options, executable_path=r'/Users/ws_fingear/Documents/workspace/chromedriver')
chrome_options = Options()
browser.get(bloomberg_frontpage_url)


browser.implicitly_wait(10)

try:
    time.sleep(6)
    browser.find_element_by_xpath('//*[@id="nav-bar-search-button"]').click() #Click "Search"
    time.sleep(5)
    browser.find_element_by_xpath('//*[@id="navi-search-input"]').send_keys('HSI') #Send 'HSI' to the searchbar
    time.sleep(5)
    browser.find_element_by_partial_link_text('HSI:IND').click()
    time.sleep(5)
    time.sleep(5)
    HSI_PE = browser.find_element_by_xpath('//[@id="root"]/div/div/section[3]/div[3]/div[1]/div/div[1]/div[1]/div/span[2]').text


可唔可以有人救救我？感謝大家

---

作者: pchai    時間: 2021-7-14 16:58

本帖最後由 pchai 於 2021-7-14 17:03 編輯

用正統既API Call
free既當然有限制

https://www.gugudata.com/api/details/stockhkrealtime

https://polygon.io/pricing

---

作者: javacomhk    時間: 2021-7-14 19:24

用Javascript 都可以.

[attach]2272855[/attach]

---

作者: takayo72    時間: 2021-7-15 13:48

提示: 作者被禁止或刪除 內容自動屏蔽

---

作者: javacomhk    時間: 2021-7-15 14:21

用得 Bloomberg data 一定唔係港股咁簡單啦。 Bloomberg 的 data license 好貴架，遂個 market 同你計。

---

作者: pchai    時間: 2021-7-15 15:45

---

作者: wood_0220    時間: 2021-7-16 15:55




帥兄呢個方法，連API 都好似唔駛用...

---

作者: javacomhk    時間: 2021-7-17 10:20

If you use website that has api provided, it is fast but may requires charges from the website.

If you use selenium, you are dealing with login credential and sometimes login captcha and javascript dynamic page.

If the destination page has url address, you can simply grab these static page content and put it in BeautifulSoup or json and based on the css selector print the results as described here. https://towardsdatascience.com/web-scraping-basics-82f8b5acd45c