本帖最後由 hkguy2020 於 2025-2-23 16:00 編輯
我仲諗緊點樣 code: RKLLM-Gradio, local pre-trained NPU enabled LLM and LangChain.
問過自己幾個 local llm, Llama-3.1-Instruct-8B 幾全面, 畀大家參考吓. 當然有 code 係好, 但真係 run 得到, 先至嚟料.
RKLLM Model, Llama-3.1-Instruct-8B has been initialized successfully！
==============================
I'll guide you through the process of setting up Rockchip RK3588 with RKLLM (Rockchip Large Language Model) using Gradio for local LLM inference, leveraging the Neural Processing Unit (NPU) for acceleration and integrating LangChain for programming.
**Prerequisites:**
1. **RK3588 board**: You'll need a Rockchip RK3588-based single-board computer.
2. **Ubuntu 20.04 or later**: Install Ubuntu on your RK3588 board.
3. **Python 3.x**: Ensure Python is installed on your system.
4. **LangChain and Gradio**: Install the required libraries using pip:
```bash
pip install langchain gradio transformers torch
```
5. **RKLLM model**: You can use a pre-trained LLM like T5 or BART, which are compatible with RK3588's NPU.
**Step 1: Prepare your environment**
Create a new directory for your project and navigate to it:
```bash
mkdir rk3588-lm-project
cd rk3588-lm-project
```
**Step 2: Install the required dependencies**
Update your package list and install the necessary packages:
```bash
sudo apt update && sudo apt upgrade -y
# Install OpenCL for NPU acceleration
sudo apt-get install ocl-icd-opencl-dev libopencl1
# Install CUDA (for GPU support, but not required for this example)
sudo apt-get install nvidia-cuda-toolkit
```
**Step 3: Clone the RKLLM repository**
Clone the RKLLM GitHub repository to access pre-trained models and utilities:
```bash
git clone https://github.com/Rockchip-LiMe/rkllm.git
cd rkllm
```
**Step 4: Prepare your LLM model**
You can use a pre-trained model from the `rkllm` repository or download one of your choice. For this example, we'll use the pre-trained T5-Base model:
```bash
# Download the pre-trained T5-Base model
wget https://github.com/Rockchip-LiMe ... v0.1/t5-base.tar.gz
# Extract the model files
tar -xvf t5-base.tar.gz
```
**Step 5: Install LangChain and Gradio**
Install the required libraries using pip:
```bash
pip install langchain gradio transformers torch
```
**Step 6: Create a sample project with LangChain and Gradio**
Create a new Python file (`app.py`) to define your LLM-based application:
```python
import gradio as gr
from langchain.agents import ChainAgent
from transformers import T5Tokenizer
# Load the pre-trained model and tokenizer
tokenizer = T5Tokenizer.from_pretrained('t5-base')
model = torch.load('./models/t5-base.pth', map_location='cuda')
# Define a sample function using LangChain's ChainAgent
def chat(input_text):
agent = ChainAgent(
model=model,
tokenizer=tokenizer,
max_length=1024,
min_length=64,
)
response = agent.generate_response(input_text)
return response
# Create a Gradio interface for local LLM inference
demo = gr.Interface(chat, inputs=[gr.Textbox(label="Input")], outputs=[gr.Textbox(label="Output")])
# Launch the demo
demo.launch()
```
**Step 7: Run your application**
Run your Python script using:
```bash
python app.py
```
This will launch a Gradio interface for local LLM inference, allowing you to interact with your pre-trained model.
Note that this is just a basic example to get you started. You can customize the LangChain and Gradio components to suit your specific requirements and integrate them with other AI models or libraries as needed.
--------------------------------------------------------------------------------------
Stage Total Time (ms) Tokens Time per Token (ms) Tokens per Second
--------------------------------------------------------------------------------------
Prefill 2118.11 94 22.53 44.38
Generate 281084.43 798 353.18 2.83
--------------------------------------------------------------------------------------
詳細睇 code, 唔 work. 最重要嘅 LangChain wrap RKLLM, 竟然無. ChatGPT 咁樣寫:
📌 3. Create a LangChain Wrapper for RKLLM
RKLLM runs in the terminal, so we create a custom LangChain LLM class to call it.
Create rkllm_langchain.py:
from langchain.llms.base import LLM
import subprocess
class RKLLM(LLM):
model_path: str
def _call(self, prompt: str, stop=None, run_manager=None):
# Run RKLLM inference
result = subprocess.run(["./rkllm_infer", self.model_path, prompt], capture_output=True, text=True)
return result.stdout.strip()
@property
def _identifying_params(self):
return {"model_path": self.model_path}
Test it:
from rkllm_langchain import RKLLM
rk_llm = RKLLM(model_path="llama-7b.rknn")
response = rk_llm("Tell me a joke.")
print(response)
Programming 其實係自家嘢 (當然最奇怪係 LangChain 點樣 wrap 到 RKLLM), 呢個位寫到咁多, 唔再寫囉. |
簡體版
訂閱頻道
訂閱電子報
發表於 2025-2-23 13:31
| 
