Table of Contents

This page organizes the articles of The Kaitchup into categories. It supplements Substack’s search engine (top-left corner of the website) which is also helpful if you are searching for an article on a specific AI technique or model.

Fine-tuning

LoRA and QLoRA

• Multi-GPU Fine-tuning for Llama 3.1 70B with FSDP and QLoRA

• rsQLoRA: Fine-tune Llama 3 with Higher Ranks and QLoRA

• Fine-tune Mixtral-8x7B Quantized with AQLM (2-bit) on Your GPU

• QLoRa: Fine-Tune a Large Language Model on Your GPU

• Fine-tune LLMs on Your CPU with QLoRA

• Fine-tune a Chat Model on Your Data with QLoRa

• QA-LoRA: Quantization-Aware Fine-tuning for Large Language Models

• Fine-tune Quantized Llama 2 on Your GPU with QA-LoRA

• LQ-LoRA: Jointly Fine-tune and Quantize Large Language Models

• Combine Multiple LoRA Adapters for Llama 2

• LoRA Adapters: When a Naive Merge Leads to Poor Performance

• Don't Merge Your LoRA Adapter Into a 4-bit LLM

• DoRA: Better and Faster than LoRA?

• LoftQ: Better Initialization for a Quantization-Aware LoRA

• Training, Loading, and Merging QDoRA, QLoRA, and LoftQ Adapters

• Fine-tuning LLMs with a Chat Template

• Fine-tune the Token Embeddings and the Language Modeling Head of Llama 3

• Fine-tune Tiny Adapters for Llama 3 with VeRA

• My LLM Can't Stop Generating, How to Fix It?

• Simple QLoRA Fine-tuning with Axolotl

• Function Calling: Fine-tuning Llama 3 and Qwen2 on xLAM

• Fine-tuning Base LLMs vs. Fine-tuning Their Instruct Version

Preference Optimization: DPO, IPO, ORPO, …

• Fine-tune Better Chat Models with Distilled Identity Preference Optimization (IPO)

• Fine-tune Your Own Instruct Version of Mistral 7B with Direct Preference Optimization (DPO)

• Zephyr 7B Beta: A Good Teacher Is All You Need

• A Cheap Zephyr 7B Beta: Distilled DPO on Consumer Hardware

• ORPO: Preference Optimization without the Supervised Fine-tuning (SFT) Step

Reinforcement Learning with Human Feedback (RLHF)

• Train Instruct LLMs On Your GPU with DeepSpeed Chat — Step #1: Supervised Fine-tuning

• Train Instruct LLMs On Your GPU with DeepSpeed Chat — Step #2: Training a Reward Model

• Train Instruct LLMs On Your GPU with DeepSpeed Chat — Step #3: Reinforcement Learning with Human Feedback

Optimization

• Use FlashAttention-2 for Faster Fine-tuning and Inference

Quantization

AutoRound

• Intel AutoRound: Accurate Low-bit Quantization for LLMs

Marlin

• Marlin: Nearly Ideal Inference Speed for 4-bit Models with vLLM (1k+ tokens/sec)

AQLM

• Run a 7.7x Smaller Mixtral-8x7B on Your GPU with AQLM 2-bit Quantization

• Fine-tune Mixtral-8x7B Quantized with AQLM (2-bit) on Your GPU

GPTQ

• From 16-bit to 2-bit: Finding the Best Trade-off Between Memory-Efficiency and Accuracy

• Quantization of Llama 2 with GTPQ for Fast Inference on Your Computer

• GPTQ or bitsandbytes: Which Quantization Method to Use for LLMs — Examples with Llama 2

• Quantize and Fine-tune LLMs with GPTQ Using Transformers and TRL

AWQ

• Fast and Small Llama 2 with Activation-Aware Quantization (AWQ)

• Simple, Fast, and Memory-Efficient Inference for Mistral 7B with Activation-Aware Quantization (AWQ)

bitsandbytes NF4

• Run Llama 2 Chat Models on Your Computer

ExLlama

• Run Llama 2 70B on Your GPU with ExLlamaV2

SqueezeLLM

• SqueezeLLM: Better 3-bit and 4-bit Quantization for Large Language Models

HQQ and HQQ+

• 1-bit and 2-bit Llama 3: Quantization with HQQ and Fine-tuning with HQQ+

Efficient Loading and Inference

• Serve Multiple LoRA Adapters with vLLM

• KV Cache Quantization for Memory-Efficient Inference with LLMs

• Neural Speed: Fast Inference on CPU for 4-bit Large Language Models

• GGUF Quantization for Fast and Memory-Efficient Inference on Your CPU

• vLLM: Serve Fast Mistral 7B and Llama 2 Models from Your Computer

• Speculative Decoding for Faster Inference with Mixtral-8x7B and Gemma

• unsloth: Faster and Memory-Efficient QLoRA Fine-tuning

• Device Map: Avoid Out-of-Memory Errors When Running Large Language Models

• Run Very Large Language Models on Your Computer

• Safe, Fast, and Memory Efficient Loading of LLMs with Safetensors

• High-Speed Inference with llama.cpp and Vicuna on CPU

• vLLM: PagedAttention for 24x Faster LLM Inference

• Serve Large Language Models from Your Computer with Text Generation Inference

• Deploy Your Local GPT Server With Triton

RAG

• RAG for Mistral 7B Instruct with LlamaIndex and Transformers

• Turn Llama 3 into an Embedding Model with LLM2Vec

• Train Better Llama 3 Embeddings with Simple Contrastive Learning

Pre-training

• Continue Pre-training Llama 3 and Other LLMs on Your Computer

• TinyLlama: Pre-training a Small Llama 2 from Scratch

• ReLoRa: Pre-train a Large Language Model on Your GPU

• GaLore: Full Fine-tuning on Your GPU

Merge and Mixture of Expert

• The Mayonnaise: Rank First on the Open LLM Leaderboard with TIES-Merging

• Fine-tune a Mixture of Experts on Your Computer

• From Llama 3 70B to 120B: How to Self-Augment an LLM?

• Maixtchup: Make Your Own Mixture of Experts with Mergekit

• Run Mixtral-8x7B on Consumer Hardware with Expert Offloading

• Mixtral-8x7B: Understanding and Running the Sparse Mixture of Experts by Mistral AI

Benchmarking

• GPU Benchmarking: What Is the Best GPU for LoRA, QLoRA, and Inference?

• Optimum-Benchmark: How Fast and Memory-Efficient Is Your LLM?

• Behind the OpenLLM Leaderboard: The Evaluation Harness

• Estimate the Memory Consumption of LLMs for Inference and Fine-tuning

LLM Focus

Llama 3

• The Best Quantization Methods to Run Llama 3.1 on Your GPU

• Fine-tune Llama 3 on Your Computer

• Run Llama 3 70B on Your GPU with ExLlamaV2

• Fine-tune Llama 3 70B on Your GPU with AQLM 2-bit

• Avoid Quantizing Llama 3 8B with GPTQ and Use BitsandBytes Instead

• Llama 3 405B: Can You Fine-tune It?

• Llama 3.1: Fine-tuning on Consumer Hardware — LoRA vs. QLoRA

Llama 2

• Fine-tune Llama 2 on Your Computer with QLoRa and TRL

• Llama 2 on a Budget

• Platypus: Dataset Curation and Adapters for Better Large Language Models

• What You Cannot Do With Llama 2

OpenELM

• Fine-tune Tiny Chat Models with Apple OpenELM and ORPO

Falcon

• Insights from the Falcons

• Fine-tune Falcon-7B on Your GPU with TRL and QLoRa

• Simple and Quick Fine-Tuning of Falcon Models with QLoRA

• Introduction to the Open LLM Falcon-40B: Performance, Training Data, and Architecture

• Falcon 180B: Can It Run on Your Computer?

• Can You Use the Falcon Models For Commercial Applications?

Mistral 7B

• Mistral 7B: Recipes for Fine-tuning and Quantization on Your Computer

Microsoft phi-1.5, phi-2, and phi-3

• Phi-2: A Small Model Easy to Fine-tune on Your GPU

• How to Fine-tune, Quantize, and Run Microsoft phi-1.5

• Phi-3: Fine-tuning and Quantization on Your Computer

• Fine-tune Phi-3 Medium on Your Computer

Google Gemma

• Fine-tune Gemma 2 on Your Computer with LoRA and QLoRA

• Google's Gemma: Fine-tuning, Quantization, and Inference on Your Computer

• Fine-tune a Better Google Gemma with Unsloth and Distilled DPO

Qwen

• Fine-tuning and Quantization of Qwen1.5 LLMs on Your Computer

• Qwen2 vs. Llama 3: QLoRA Learning Curves and Quantization Performance

Yi

• Yi: Fine-tune and Run One of the Best Bilingual LLMs on Your Computer

SmolLM

• SmolLM: Full Fine-tuning and Aligning Tiny LLMs on Your Computer

VLMs

• Florence-2: Run Multitask Vision-language Models on Your Computer

• Fine-tune a Multimodal Chat Model with Florence-2 on Your Computer

Machine Translation

Fine-tuning

• Llama 2 MT: Turn Llama 2 into a Translation System with QLoRA

GPT

• GPT-3.5 Translates Paragraphs Better

• Translate with ChatGPT

• Translate with GPT-3

Evaluation

• BLEU: A Misunderstood Metric from Another Age

• 12 Critical Flaws of BLEU

• Traditional Versus Neural Metrics for Machine Translation Evaluation

• Scientific Credibility in Machine Translation Research: Pitfalls and Promising Trends

• How Good Is Google PaLM at Translation?

Data Processing

• Datasets to Train, Validate, and Evaluate Machine Translation

• Data Preprocessing for Machine Translation