hkguy2020

尋日收到 Ras pi 5 16gb, 即試 Qwen 14b (8gb), load 到行到, performance 可以接受, 但 youtube 見 Orange Pi  5 係快得多. 行大嘅 model, 答嘅嘢係堅料好多. 之後緊係要試 local 行 DeepSeek, 聽到係抄, 集大成. 無寄望亦無失望, 問:

'write a Flutter ecommerce shopping cart app. Use Bloc for state management and use clean architecture. Features include di injectable, repository and use case for the repository. Use dio to communicate with the Strapi backend, product catalogue with different categories, user registration and login, Stripe credit card payment gateway, user can toggle light and dark theme by using adaptive_theme package. Please provide a complete program and I need to reference every program code in clean architecture structure. Thank you very much.'

唔至到唔識答, 但只係答大路嘢, 最後少少唔在行嘅 programming 搭番. 呢個 programmer 真係呃飯食, 我畀20分, 同 Qwen 比, 人哋真係識寫個完整 app 出嚟.

依家試緊 ollama run --verbose llama3.3, pull 到 28% (12gb/42gb), 行到機會不大.

下載 (36.05 KB)

2025-1-30 10:06

zero-lam

好似好勁咁，
驚遲d 畀人淘汰，想學下ai,
但無從入手

利申，淨係識砌下機

hkguy2020

收到 Orange Pi 5 Max 16gb 都有兩個禮拜 (之間, 我個 Rapi 4 CMS server load 唔起, 可能同 Ubuntu 24.10 有關, 好傷) , set up 好, 用來 local 行 llm, 都搞掂. 喺 Orange Pi 係要用 rkllm (https://github.com/c0zaut/RKLLM-Gradio, 呢個最簡單), 用 rkllm, 係會用到 Orange Pi 上嘅 NPU (6 TOPS), 比 Raspi 5 快得多, 但 developer 唔多, 依家popular 都係講用 Ollama.

Ollama 去夾 Langchain 有 Python library. 而喺 Orange Pi 5, 用 rkllm 係好麻煩, llm 要先 convert to w8a8 format (https://huggingface.co/Pelochus/ezrkllm-collection). 要夾 Langchain, 未搵到 library, 我要慢慢研究.

今晚搵到個要買嘅 resource 好好, 好 practical. 我諗照住學, 可以放係 Raspi 5 做 AI projects:
2025 Master Langchain and Ollama - Chatbot, RAG and Agents (如想學 Ollama, Langchan, RAG ..., 自己 google 吓).

下載 (149.48 KB)

2025-2-22 06:55

cadelite

回覆 13# hkguy2020

CHing，請教一下 用 Mini PC (例如 : M*refine/GMKT*c) 來做同樣嘅事會唔會比較快，有效率得多?

本身都有 RPI 5，不過發現佢好多軟件都未必支援，或者要專登為佢 compile 過，已經氣餒

如果行 AI，開放 AI agent/bot 俾人用，唔知有無可能? 單純開放做 webhosting 可能無乜人會理會

hkguy2020

回覆  hkguy2020

CHing，請教一下 用 Mini PC (例如 : M*refine/GMKT*c) 來做同樣嘅事會唔會比較快，有效 ...
cadelite 發表於 2025-2-21 17:15

local 行 7-8b llm (qwen, gemma, llama, phi, deepseek), 有 npu 會好快, 感覺同外面大 server 差不多, 同好在係 local, 可以 customize llm. Mini pc 應該係無 npu, 而 gpu 都係弱, 所以 Orange Pi 5 好玩, Raspi Pi 5 一有 Hailo-10H 後, 會好惦. Hailo-10H 都無得貴 (Raspi Pi 5 16gb 大約US$170, >> NVIDIA Jetson Orin Nano 8gb 要超過 US$600 <<).

下載 (104 KB)

2025-2-22 03:30

依家, Raspi 5 我覺得好好 (Orange Pi 5 一樣乜都做到), 勤力去搵料, 再加自己功力已可以. 加埋搵 AI 幫手, 就無得頂. AI 喺 Linux 上 development, 易搵料好多, 都好成熟.

好快將來, software development 無 AI, 搵唔到食. 搵食以 cross platform app with AI (估 Flutter 佔 market 最少8成, 遲啲 Flutter 喺 web 再掂啲, react native 就再無 market), 再加埋 headless cms with AI (backend, Linux deployment), 再差環境應該都搵到食. 自己 provide AI Agent ..... 無得同大 server 撼喎.

cadelite

回覆 15# hkguy2020

唔知係唔係 AI 又炒起咗，RPI5 嘅 AI Halio Hat Kit 又炒到過千銀，比 RPI5 仲要貴 (16GB 版本除外)

買埋週邊產品，花費可能比得上一部 M4 Mini

hkguy2020

睇完後, 覺得 LangChain 同 NPU 無關. NPU 只係加快 llm process. LangChain development 喺 Ras Pi 同 Orange Pi, 應該無分別 (當然, Orange Pi 要用 rkllm).

ChatGPT gives summary:
LangChain doesn't directly interface with NPUs, but you can potentially use LangChain with models that are deployed on hardware with NPU acceleration. LangChain is mostly about structuring language model-based workflows, while NPUs are more focused on accelerating the underlying computations. If you're using an NPU in a setup with LangChain, the NPU would likely be involved in speeding up the inference of your language model, but LangChain itself would handle the logic and chaining of queries and responses.

hkguy2020

回覆  hkguy2020

唔知係唔係 AI 又炒起咗，RPI5 嘅 AI Halio Hat Kit 又炒到過千銀，比 RPI5 仲要貴 (16G ...
cadelite 發表於 2025-2-22 14:27

Hailo-8L vs Hailo-10H:
Hailo-8L 唔會加快 llm, 只可以用喺 imaging .... Hailo-10H 可以加快 llm, 但要等到今年第一季尾先出. (唔知有幾多貨? 會唔會炒起?)

想即刻 llm 行得快 (40 tokens per second), 可考慮 Orange Pi 5 Max 16gb  -- 有 NPU(6 TOPS), can mount SSD directly, LPDDR5, 快過, 價平過 Ras Pi 5, build solid, 可以用 Ras Pi 5 power supply (Orange Pi 5 係 4A) . 唔好處: need very skillful to set up the rkllm.

要 programming AI, 一定要熟 Python.

caihk

Hailo-8L vs Hailo-10H:
Hailo-8L 唔會加快 llm, 只可以用喺 imaging .... Hailo-10H 可以加快 llm, 但 ...
hkguy2020 發表於 2025-2-22 15:55

睇返啲有限資料，Hailo-10H之所以會快，係因爲張卡有ddr記憶體。不過10H好似只支換llm，tts（text to image），而用唔到語音轉文字，文字生成影片/音樂，雖然張算力卡係pcie，可以插係任何 pc / sbc 裝置，但係感覺吸引力唔係咁高。

opi5似乎係平價ai板既不錯選擇，大約港紙1千左右就買到，係算力方面當然同n卡由上百Tops起跳係無得比，價錢亦同樣係無得比。睇啲評價係好過nanapi，radxa呢啲，主要差別在於軟件支換。不過睇啲rk3588教學，rockchip用起上黎都幾麻煩， 要將native模型 -----〉轉成onnx格式 ----〉rknn 格式，無計皆因目前啲模型係用gpu訓練 （即係用cuda），而唔係針對npu寫。不過啲模型可以i換onnx格式係好事，咁 amd 同 intel 機都用得，唔洗局限要買nvidia顯卡（先可以run得快）。

依家ai興起，買tops唔平，所以本地部署都係化算既。希望快啲有國產平價算力卡出現，普及化ai，等一般消費者都可以享受到。

hkguy2020

睇返啲有限資料，Hailo-10H之所以會快，係因爲張卡有ddr記憶體。不過10H好似只支換llm，tts（text to ima ...
caihk 發表於 2025-2-23 08:38

喺 Hailo 官網, 有咁講:

At CES 2025, we debuted the Hailo-10 GenAI accelerator, designed to run large language models (LLMs) and generative AI directly on edge devices. Demonstrating Stable Diffusion 1.5 offline, Hailo-10 transforms text into high-quality images with ultra-low latency, enhanced privacy, and seamless integration—all without cloud reliance.

Hailo-10H demonstrate Stable Diffusion

似乎好快會有得賣.