交流及求教: 有關 Notebook's performance (Thinkpad)

wongty

最近有個 Project 要用 Powershell process a big XML file (~300M), the XML structure is simple, 就係好多 records (~400K records). Basically it just converts the XML to CSV file, simple text processing. 我自己用開 Thinkpad W520 (i7 2860QM), 寫完個script竟然要行成個鐘.  再breakdown the script, found that it used 25 min to read the XML file to Powershell internal variable ([XML]$xml = Get-Content ' .\input.XML'), and then use ~30 min to write it to CSV file line by line. 咁我心唸部 W520 成十年前出, 係咪時候退休呢?

於是搵咗部 Dell M4800 (i7 4810MQ) 嚟試, 唸住如果快好多, 就轉戶過去, 但結果係 -- 行咗 1 hr 17 min, reading XML is slower (~45min) & processing data is level (~30 min). 有 D 失望.

於是再試吓部 X395 (Ryzen 3500u) that I just bought for my family. I had lot of expectation before I run the script, and thought I would buy T495 (做緊特價 $52xx) for myself if the result is satisfactory.  Unfortunately the result is astonishing -- It took over 3 hours to complete the script!

All 3 notebooks are 16G & running SSD, and they are all 4-Core 8-Thread. In my W520 & M4800, when reading XML file, CPU ~25%, yet 16G is marginally enough as it ate up all memory when it read the XML with (disk activities was low, mostly 0-15%). Processing record was not heavily loaded (CPU ~20%, RAM ~40%, Disk ~15%). In X395, more disk activities is found (15G RAM usable, 1G is shared by GPU).

我知呢度好多高手, 究竟有冇D tuning 可以試 (esp X395 and M4800)? , 定係其實D新CPU嘅processing power 真係冇點進步呢?

補充: all notebooks are Win 10, and power plugged during the tests.

victor59hk

有無做sorting ?
csv 做好 sorting , 開嗰時會快好多

free3sky

試下ramdisk ,放入面搞
看看會唔會快左
似乎大部分用係存取到

KinChungE

Excel唔少task係single threaded
對single threaded既task, 唔好諗住新機有咩大期望
新U都係一路提升多核performance, 單核最多每年幾個%咁上下

至於AMD行Office慢, 個人感覺的確如此
我試過i5-7200u用落爽過Ryzen 5 3500u, 但係冇實際跑benchmark

mre

唔好一次過讀完成個300MB file再處理，試下逐隻record讀完寫完先處理下一隻。

via HKEPC IR Pro 3.6.1 - Android(3.3.1)

zore0083

的確近年來 CPU 都是提升多核為主。
另外windows 在處理文字轉換格式上是比較差。
以前我其中一間舊公司都是要 XML/ Excel convert to CSV 500MB左右要2-3小時 (用 windows server 2003)。
但當我轉另一間公司都是 XML convert to CSV 1GB 左右用Linux (Redhat) 半小時左右。
我做過的電信公司及金融機構都是用Linux / Unix convert raw data to CSV。
所以我相信要處理大量的文字轉換格式應該用 Linux / Unix 比較有效率。

wongty

多謝各位師兄, 其實我個 Powershell script 係做 XML 轉去 CSV. 個 XML file 係 300MB左右, 開 Powershell 剩係呢一句已經行咗 25 min (最快嘅W520):

1. > [XML]$xml = Get-Content './input.XML'

複製代碼

其間disk activities 好少, 所以 RAM disk 或者 sorting 都應該冇關係, 我其實最奇怪係剩行呢句用嘅時間竟然係:
W520 (i7-2860):      25min
M4800 (i7-4810):    45min
X395 (Ryzen 3500u):   100min

各位師兄如果隨手搵個幾百MB嘅XML file 都可以開個 Powershell 試吓上面個 command, 睇吓要行幾耐.

暴力乂狗仔

多謝各位師兄, 其實我個 Powershell script 係做 XML 轉去 CSV. 個 XML file 係 300MB左右, 開 Powershell  ...
wongty 發表於 2020-4-11 00:27

    Windows 對AMD 優化重要一啲時間....
    我X395都係LAG, 但更新Windows / Office 一次就好一次...

WHITERICE

有無試過set Priority of Running 去到最高黎RUN, Powershell 好多時唔想影響電腦正常運作,只係攞好少資源去行.