Spaces:
Sleeping
Sleeping
File size: 28,316 Bytes
a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 33a1d2e a79bc8f ae07f77 2326498 24107f3 2326498 24107f3 a79bc8f ae07f77 2326498 f0033ab 2326498 f0033ab 2326498 a79bc8f 4be72e0 a79bc8f 24107f3 ae07f77 a79bc8f 4be72e0 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 24107f3 ae07f77 24107f3 ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f 2326498 ae07f77 2326498 ecf6a69 022b2da ecf6a69 022b2da ecf6a69 022b2da ecf6a69 022b2da e08f8c4 cef8ecd e08f8c4 ae07f77 cef8ecd ae07f77 e08f8c4 ae07f77 2326498 ae07f77 2326498 ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 24107f3 a79bc8f 24107f3 4be72e0 24107f3 4be72e0 24107f3 a79bc8f 24107f3 ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f ae07f77 a79bc8f 2dff966 808203f d4bc333 808203f d4bc333 808203f 2dff966 24107f3 2dff966 a79bc8f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 |
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Router Models AWQ Quantization with LLM Compressor (vLLM Native)\n",
"\n",
"This notebook quantizes the CourseGPT-Pro router models to AWQ (Activation-aware Weight Quantization) format using **LLM Compressor** - vLLM's native quantization tool.\n",
"\n",
"**Models to quantize:**\n",
"- `Alovestocode/router-gemma3-merged` (27B)\n",
"- `Alovestocode/router-qwen3-32b-merged` (33B)\n",
"\n",
"**Output:** AWQ-quantized models ready for vLLM inference with optimal performance.\n",
"\n",
"**Why LLM Compressor?**\n",
"- Native vLLM integration (better compatibility)\n",
"- Supports advanced features (pruning, combined modifiers)\n",
"- Actively maintained by vLLM team\n",
"- Optimized for vLLM inference engine\n",
"\n",
"**⚠️ IMPORTANT:** If you see errors about `AWQModifier` parameters, **restart the kernel** (Runtime → Restart runtime) and run all cells from the beginning. The notebook uses `AWQModifier()` without parameters (default 4-bit AWQ).\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install Dependencies\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Install required packages\n",
"# LLM Compressor is vLLM's native quantization tool\n",
"# Note: Package name is 'llmcompressor' (no hyphen), may need to install from GitHub\n",
"%pip install -q transformers accelerate huggingface_hub\n",
"%pip install -q torch --index-url https://download.pytorch.org/whl/cu118\n",
"\n",
"# Try installing llmcompressor from PyPI first, fallback to GitHub if not available\n",
"try:\n",
" import llmcompressor\n",
" print(\"✅ llmcompressor already installed\")\n",
"except ImportError:\n",
" print(\"Installing llmcompressor...\")\n",
" # Try PyPI first\n",
" import subprocess\n",
" import sys\n",
" result = subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"llmcompressor\"], \n",
" capture_output=True, text=True)\n",
" if result.returncode != 0:\n",
" # Fallback to GitHub installation\n",
" print(\"PyPI installation failed, trying GitHub...\")\n",
" subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \n",
" \"git+https://github.com/vllm-project/llm-compressor.git\"], \n",
" check=False)\n",
" print(\"✅ llmcompressor installed\")\n",
"\n",
"# Utility function to check disk space\n",
"import shutil\n",
"def check_disk_space():\n",
" \"\"\"Check available disk space.\"\"\"\n",
" total, used, free = shutil.disk_usage(\"/\")\n",
" print(f\"Disk Space: {free / (1024**3):.2f} GB free out of {total / (1024**3):.2f} GB total\")\n",
" return free / (1024**3) # Return free space in GB\n",
"\n",
"print(\"Initial disk space:\")\n",
"check_disk_space()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Authenticate with Hugging Face\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from huggingface_hub import login\n",
"import os\n",
"\n",
"# Login to Hugging Face (you'll need a token with write access)\n",
"# Get your token from: https://huggingface.co/settings/tokens\n",
"HF_TOKEN = \"your_hf_token_here\" # Replace with your token\n",
"\n",
"login(token=HF_TOKEN)\n",
"os.environ[\"HF_TOKEN\"] = HF_TOKEN\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Configuration\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Model configurations\n",
"MODELS_TO_QUANTIZE = {\n",
" \"router-gemma3-merged\": {\n",
" \"repo_id\": \"Alovestocode/router-gemma3-merged\",\n",
" \"output_repo\": \"Alovestocode/router-gemma3-merged-awq\", # Or keep same repo\n",
" \"model_type\": \"gemma\",\n",
" },\n",
" \"router-qwen3-32b-merged\": {\n",
" \"repo_id\": \"Alovestocode/router-qwen3-32b-merged\",\n",
" \"output_repo\": \"Alovestocode/router-qwen3-32b-merged-awq\", # Or keep same repo\n",
" \"model_type\": \"qwen\",\n",
" }\n",
"}\n",
"\n",
"# AWQ quantization config\n",
"AWQ_CONFIG = {\n",
" \"w_bit\": 4, # 4-bit quantization\n",
" \"q_group_size\": 128, # Group size for quantization\n",
" \"zero_point\": True, # Use zero-point quantization\n",
" \"version\": \"GEMM\", # GEMM kernel (better for longer contexts)\n",
"}\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Quantization Function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# LLM Compressor (vLLM native quantization tool)\n",
"# Import with error handling in case installation failed\n",
"try:\n",
" from llmcompressor import oneshot\n",
" # Correct import path: AWQModifier is in modifiers.awq, not modifiers.quantization\n",
" from llmcompressor.modifiers.awq import AWQModifier\n",
" LLM_COMPRESSOR_AVAILABLE = True\n",
" print(\"✅ LLM Compressor imported successfully\")\n",
"except ImportError as e:\n",
" print(f\"❌ Failed to import llmcompressor: {e}\")\n",
" print(\"Please ensure llmcompressor is installed:\")\n",
" print(\" %pip install llmcompressor\")\n",
" print(\" OR\")\n",
" print(\" %pip install git+https://github.com/vllm-project/llm-compressor.git\")\n",
" print(\"\\nNote: If import still fails, try:\")\n",
" print(\" %pip install --upgrade llmcompressor\")\n",
" LLM_COMPRESSOR_AVAILABLE = False\n",
" raise\n",
"\n",
"from transformers import AutoTokenizer\n",
"from huggingface_hub import HfApi, scan_cache_dir, upload_folder\n",
"import torch\n",
"import shutil\n",
"import gc\n",
"import os\n",
"\n",
"# Try to import delete_revisions (may not be available in all versions)\n",
"try:\n",
" from huggingface_hub import delete_revisions\n",
" DELETE_REVISIONS_AVAILABLE = True\n",
"except ImportError:\n",
" # delete_revisions might not be available, we'll use alternative method\n",
" DELETE_REVISIONS_AVAILABLE = False\n",
" print(\"Note: delete_revisions not available, will use alternative cache cleanup method\")\n",
"\n",
"def quantize_model_to_awq(\n",
" model_name: str,\n",
" repo_id: str,\n",
" output_repo: str,\n",
" model_type: str,\n",
" awq_config: dict,\n",
" calibration_dataset_size: int = 128\n",
"):\n",
" \"\"\"Quantize a model to AWQ format using LLM Compressor (vLLM native).\n",
" \n",
" Args:\n",
" model_name: Display name for the model\n",
" repo_id: Source Hugging Face repo ID\n",
" output_repo: Destination Hugging Face repo ID\n",
" model_type: Model type (gemma/qwen) for tokenizer selection\n",
" awq_config: AWQ quantization configuration\n",
" calibration_dataset_size: Number of calibration samples\n",
" \"\"\"\n",
" print(f\"\\n{'='*60}\")\n",
" print(f\"Quantizing {model_name} with LLM Compressor (vLLM native)\")\n",
" print(f\"Source: {repo_id}\")\n",
" print(f\"Destination: {output_repo}\")\n",
" print(f\"{'='*60}\\n\")\n",
" \n",
" # Check disk space before starting\n",
" free_space_before = check_disk_space()\n",
" if free_space_before < 30:\n",
" print(f\"⚠️ WARNING: Low disk space ({free_space_before:.2f} GB). Quantization may fail.\")\n",
" \n",
" # Step 1: Create temporary output directory\n",
" import tempfile\n",
" temp_output_dir = f\"./temp_{model_name.replace('-', '_')}_awq\"\n",
" print(f\"[1/4] Creating temporary output directory: {temp_output_dir}\")\n",
" os.makedirs(temp_output_dir, exist_ok=True)\n",
" \n",
" # Step 2: Prepare calibration dataset\n",
" print(f\"\\n[2/4] Preparing calibration dataset ({calibration_dataset_size} samples)...\")\n",
" \n",
" # Create calibration dataset for router agent\n",
" calibration_texts = [\n",
" \"You are the Router Agent coordinating Math, Code, and General-Search specialists.\",\n",
" \"Emit EXACTLY ONE strict JSON object with keys route_plan, route_rationale, expected_artifacts,\",\n",
" \"Solve a quadratic equation using Python programming.\",\n",
" \"Implement a binary search algorithm with proper error handling.\",\n",
" \"Explain the concept of gradient descent in machine learning.\",\n",
" \"Write a function to calculate the Fibonacci sequence recursively.\",\n",
" \"Design a REST API endpoint for user authentication.\",\n",
" \"Analyze the time complexity of merge sort algorithm.\",\n",
" ]\n",
" \n",
" # Repeat to reach desired size\n",
" while len(calibration_texts) < calibration_dataset_size:\n",
" calibration_texts.extend(calibration_texts[:calibration_dataset_size - len(calibration_texts)])\n",
" \n",
" calibration_texts = calibration_texts[:calibration_dataset_size]\n",
" print(f\"✅ Calibration dataset prepared: {len(calibration_texts)} samples\")\n",
" \n",
" # Step 3: Quantize model using LLM Compressor\n",
" print(f\"\\n[3/4] Quantizing model to AWQ with LLM Compressor (this may take 30-60 minutes)...\")\n",
" print(f\"Config: {awq_config}\")\n",
" print(\"⚠️ LLM Compressor will load the model, quantize it, and save to local directory\")\n",
" \n",
" if not LLM_COMPRESSOR_AVAILABLE:\n",
" raise ImportError(\"LLM Compressor is not available. Please install it first.\")\n",
" \n",
" try:\n",
" # LLM Compressor's oneshot function handles everything:\n",
" # - Loading the model\n",
" # - Quantization with calibration data\n",
" # - Saving quantized model\n",
" print(f\" → Starting quantization with LLM Compressor...\")\n",
" print(f\" → This may take 30-60 minutes depending on model size...\")\n",
" \n",
" # AWQModifier requires quantization_config with config_groups\n",
" # Create quantization config for 4-bit AWQ\n",
" from compressed_tensors.quantization import QuantizationConfig, BaseQuantizationConfig\n",
" \n",
" print(f\" → Creating quantization config for 4-bit AWQ...\")\n",
" # QuantizationConfig requires config_groups - a dict mapping layer names to configs\n",
" # For AWQ, we use a default config group that applies to all layers\n",
" quant_config = QuantizationConfig(\n",
" config_groups={\n",
" \"default\": BaseQuantizationConfig(\n",
" num_bits=4, # 4-bit quantization\n",
" group_size=128, # Group size\n",
" zero_point=True # Zero-point quantization\n",
" )\n",
" }\n",
" )\n",
" \n",
" print(f\" → Creating AWQModifier with quantization config...\")\n",
" modifiers = [AWQModifier(quantization_config=quant_config)]\n",
" print(f\" → AWQModifier created successfully\")\n",
" \n",
" # Call oneshot with the modifier\n",
" print(f\" → Starting quantization process...\")\n",
" oneshot(\n",
" model=repo_id,\n",
" output_dir=temp_output_dir,\n",
" modifiers=modifiers,\n",
" token=os.environ.get(\"HF_TOKEN\"),\n",
" # Calibration data: list of strings\n",
" calibration_data=calibration_texts[:min(calibration_dataset_size, 128)]\n",
" )\n",
" \n",
" print(f\"✅ Model quantized to AWQ successfully\")\n",
" except Exception as e:\n",
" print(f\"❌ Quantization failed: {e}\")\n",
" print(f\"\\nTroubleshooting:\")\n",
" print(f\"1. Ensure llmcompressor is installed: %pip install llmcompressor\")\n",
" print(f\"2. Or install from GitHub: %pip install git+https://github.com/vllm-project/llm-compressor.git\")\n",
" print(f\"3. Check that you have sufficient GPU memory (40GB+ recommended)\")\n",
" import traceback\n",
" traceback.print_exc()\n",
" raise\n",
" \n",
" # Step 4: Upload to Hugging Face\n",
" print(f\"\\n[4/4] Uploading quantized model to {output_repo}...\")\n",
" \n",
" # Create repo if it doesn't exist\n",
" api = HfApi()\n",
" try:\n",
" api.create_repo(\n",
" repo_id=output_repo,\n",
" repo_type=\"model\",\n",
" exist_ok=True,\n",
" token=os.environ.get(\"HF_TOKEN\")\n",
" )\n",
" print(f\"✅ Repository ready: {output_repo}\")\n",
" except Exception as e:\n",
" print(f\"Note: Repo may already exist: {e}\")\n",
" \n",
" # Upload the quantized model directory\n",
" try:\n",
" upload_folder(\n",
" folder_path=temp_output_dir,\n",
" repo_id=output_repo,\n",
" repo_type=\"model\",\n",
" token=os.environ.get(\"HF_TOKEN\"),\n",
" ignore_patterns=[\"*.pt\", \"*.bin\"] # Only upload safetensors\n",
" )\n",
" print(f\"✅ Quantized model uploaded to {output_repo}\")\n",
" except Exception as e:\n",
" print(f\"❌ Upload failed: {e}\")\n",
" import traceback\n",
" traceback.print_exc()\n",
" raise\n",
" \n",
" # Step 5: Clean up to free disk space (critical for Colab)\n",
" print(f\"\\n[5/5] Cleaning up local files to free disk space...\")\n",
" \n",
" # Delete temporary output directory\n",
" try:\n",
" import shutil\n",
" shutil.rmtree(temp_output_dir)\n",
" print(f\" ✅ Deleted temporary directory: {temp_output_dir}\")\n",
" except Exception as e:\n",
" print(f\" ⚠️ Could not delete temp directory: {e}\")\n",
" \n",
" # Free GPU memory\n",
" torch.cuda.empty_cache()\n",
" gc.collect()\n",
" \n",
" # Clear Hugging Face cache for the source model (frees ~50-70GB)\n",
" print(f\" → Clearing Hugging Face cache for {repo_id}...\")\n",
" try:\n",
" cache_info = scan_cache_dir()\n",
" # Find and delete revisions for the source model\n",
" revisions_to_delete = []\n",
" for repo in cache_info.revisions:\n",
" if repo.repo_id == repo_id:\n",
" revisions_to_delete.append(repo)\n",
" \n",
" if revisions_to_delete:\n",
" if DELETE_REVISIONS_AVAILABLE:\n",
" # Use delete_revisions if available\n",
" delete_revisions(revisions_to_delete)\n",
" print(f\" ✅ Deleted {len(revisions_to_delete)} cached revision(s) for {repo_id}\")\n",
" else:\n",
" # Alternative: Delete cache directories manually\n",
" deleted_count = 0\n",
" for revision in revisions_to_delete:\n",
" try:\n",
" # Get the cache directory path\n",
" cache_path = revision.snapshot_path if hasattr(revision, 'snapshot_path') else None\n",
" if cache_path and os.path.exists(cache_path):\n",
" shutil.rmtree(cache_path)\n",
" deleted_count += 1\n",
" except Exception as e:\n",
" print(f\" ⚠️ Could not delete {revision.repo_id}: {e}\")\n",
" \n",
" if deleted_count > 0:\n",
" print(f\" ✅ Deleted {deleted_count} cached revision(s) for {repo_id}\")\n",
" else:\n",
" print(f\" ℹ️ Found {len(revisions_to_delete)} cached revision(s) but couldn't delete them\")\n",
" print(f\" Try manually: huggingface-cli scan-cache --dir ~/.cache/huggingface\")\n",
" else:\n",
" print(f\" ℹ️ No cached revisions found for {repo_id}\")\n",
" except Exception as e:\n",
" print(f\" ⚠️ Cache cleanup warning: {e} (continuing...)\")\n",
" print(f\" You can manually clean cache with: huggingface-cli scan-cache\")\n",
" \n",
" # Check disk space after cleanup\n",
" free_space_after = check_disk_space()\n",
" print(f\"\\n✅ Cleanup complete! Free space: {free_space_after:.2f} GB\")\n",
" \n",
" print(f\"\\n✅ {model_name} quantization complete!\")\n",
" print(f\"Model available at: https://huggingface.co/{output_repo}\")\n",
" print(f\"💾 Local model files deleted to save disk space\")\n",
" print(f\"🚀 Model is ready for vLLM inference with optimal performance!\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"quantize_model_to_awq(\n",
" model_name=\"Router-Gemma3-27B\",\n",
" repo_id=MODELS_TO_QUANTIZE[\"router-gemma3-merged\"][\"repo_id\"],\n",
" output_repo=MODELS_TO_QUANTIZE[\"router-gemma3-merged\"][\"output_repo\"],\n",
" model_type=MODELS_TO_QUANTIZE[\"router-gemma3-merged\"][\"model_type\"],\n",
" awq_config=AWQ_CONFIG,\n",
" calibration_dataset_size=128\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Quantize Router-Qwen3-32B-Merged\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"quantize_model_to_awq(\n",
" model_name=\"Router-Qwen3-32B\",\n",
" repo_id=MODELS_TO_QUANTIZE[\"router-qwen3-32b-merged\"][\"repo_id\"],\n",
" output_repo=MODELS_TO_QUANTIZE[\"router-qwen3-32b-merged\"][\"output_repo\"],\n",
" model_type=MODELS_TO_QUANTIZE[\"router-qwen3-32b-merged\"][\"model_type\"],\n",
" awq_config=AWQ_CONFIG,\n",
" calibration_dataset_size=128\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Verify Quantized Models\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Verify quantized models with vLLM (recommended) or Transformers\n",
"from transformers import AutoTokenizer\n",
"\n",
"def verify_awq_model_vllm(repo_id: str):\n",
" \"\"\"Verify AWQ model can be loaded with vLLM (recommended).\"\"\"\n",
" print(f\"\\nVerifying {repo_id} with vLLM...\")\n",
" \n",
" try:\n",
" # Try importing vLLM\n",
" try:\n",
" from vllm import LLM, SamplingParams\n",
" except ImportError:\n",
" print(\"⚠️ vLLM not available, skipping vLLM verification\")\n",
" return False\n",
" \n",
" # Load with vLLM (auto-detects AWQ)\n",
" llm = LLM(\n",
" model=repo_id,\n",
" quantization=\"awq\",\n",
" trust_remote_code=True,\n",
" token=os.environ.get(\"HF_TOKEN\"),\n",
" gpu_memory_utilization=0.5 # Lower for verification\n",
" )\n",
" \n",
" # Test generation\n",
" sampling_params = SamplingParams(\n",
" temperature=0.0,\n",
" max_tokens=10\n",
" )\n",
" \n",
" test_prompt = \"You are the Router Agent. Test prompt.\"\n",
" outputs = llm.generate([test_prompt], sampling_params)\n",
" \n",
" generated_text = outputs[0].outputs[0].text\n",
" print(f\"✅ vLLM loads and generates correctly\")\n",
" print(f\"Generated: {generated_text[:100]}...\")\n",
" \n",
" del llm\n",
" torch.cuda.empty_cache()\n",
" \n",
" return True\n",
" except Exception as e:\n",
" print(f\"❌ vLLM verification failed: {e}\")\n",
" import traceback\n",
" traceback.print_exc()\n",
" return False\n",
"\n",
"def verify_awq_model_transformers(repo_id: str):\n",
" \"\"\"Verify AWQ model can be loaded with Transformers (fallback).\"\"\"\n",
" print(f\"\\nVerifying {repo_id} with Transformers...\")\n",
" \n",
" try:\n",
" # Load tokenizer\n",
" tokenizer = AutoTokenizer.from_pretrained(\n",
" repo_id,\n",
" trust_remote_code=True,\n",
" token=os.environ.get(\"HF_TOKEN\")\n",
" )\n",
" \n",
" # Try loading with AutoAWQ (if available)\n",
" try:\n",
" from awq import AutoAWQForCausalLM\n",
" model = AutoAWQForCausalLM.from_quantized(\n",
" repo_id,\n",
" fuse_layers=True,\n",
" trust_remote_code=True,\n",
" device_map=\"auto\",\n",
" token=os.environ.get(\"HF_TOKEN\")\n",
" )\n",
" \n",
" # Test generation\n",
" test_prompt = \"You are the Router Agent. Test prompt.\"\n",
" inputs = tokenizer(test_prompt, return_tensors=\"pt\").to(model.device)\n",
" \n",
" with torch.inference_mode():\n",
" outputs = model.generate(\n",
" **inputs,\n",
" max_new_tokens=10,\n",
" do_sample=False\n",
" )\n",
" \n",
" generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
" print(f\"✅ Transformers loads and generates correctly\")\n",
" print(f\"Generated: {generated_text[:100]}...\")\n",
" \n",
" del model\n",
" del tokenizer\n",
" torch.cuda.empty_cache()\n",
" \n",
" return True\n",
" except ImportError:\n",
" print(\"⚠️ AutoAWQ not available, skipping Transformers verification\")\n",
" return False\n",
" except Exception as e:\n",
" print(f\"❌ Transformers verification failed: {e}\")\n",
" import traceback\n",
" traceback.print_exc()\n",
" return False\n",
"\n",
"# Verify both models (prefer vLLM)\n",
"for model_key, model_info in MODELS_TO_QUANTIZE.items():\n",
" print(f\"\\n{'='*60}\")\n",
" print(f\"Verifying {model_key}\")\n",
" print(f\"{'='*60}\")\n",
" \n",
" # Try vLLM first (recommended)\n",
" vllm_ok = verify_awq_model_vllm(model_info[\"output_repo\"])\n",
" \n",
" # Fallback to Transformers if vLLM not available\n",
" if not vllm_ok:\n",
" verify_awq_model_transformers(model_info[\"output_repo\"])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notes\n",
"\n",
"- **GPU Required**: This quantization requires a GPU with at least 40GB VRAM (A100/H100 recommended)\n",
"- **Time**: Each model takes approximately 30-60 minutes to quantize\n",
"- **Disk Space**: \n",
" - Colab has limited disk space (~80GB free)\n",
" - Each source model is ~50-70GB (BF16)\n",
" - Quantized models are ~15-20GB (AWQ 4-bit)\n",
" - **The notebook automatically deletes source models after quantization to save space**\n",
"- **Cleanup**: After each model is quantized and uploaded:\n",
" - GPU memory is freed\n",
" - Hugging Face cache for source model is cleared\n",
" - Disk space is checked before/after\n",
"- **Output Repos**: Models are saved to new repos with `-awq` suffix\n",
"- **Usage**: After quantization, update your `app.py` to use the AWQ repos:\n",
" ```python\n",
" MODELS = {\n",
" \"Router-Gemma3-27B-AWQ\": {\n",
" \"repo_id\": \"Alovestocode/router-gemma3-merged-awq\",\n",
" \"quantization\": \"awq\"\n",
" },\n",
" \"Router-Qwen3-32B-AWQ\": {\n",
" \"repo_id\": \"Alovestocode/router-qwen3-32b-merged-awq\",\n",
" \"quantization\": \"awq\"\n",
" }\n",
" }\n",
" ```\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
|