Hugging Face and Commercial APIs: Mapping the Model Ecosystem

After completing this topic

If you've already built a neural network from scratch using PyTorch in Part 12, this section will teach you how to leverage the hundreds of thousands of pre-trained models that already exist today. From the Hugging Face Model Hub and Transformers library to commercial APIs from OpenAI, Anthropic, and Google, and even local serving with Ollama and vLLM. This section maps out the landscape of tools that today's AI developers must navigate.

This is the second part of the Tools Phase 3. It will serve as a practical foundation for coding agents (Claude Code, Cursor) in Part 14 and bio integration in Part 15.

Three Paths to Using Models

Here are the three main ways to use LLMs and transformers today.

Path A: Commercial APIs. Calling APIs from companies like OpenAI (GPT), Anthropic (Claude), and Google (Gemini).

• Pros: Immediate access to top-tier models, no need to worry about infrastructure, automatic updates.
• Cons: Costs, data privacy concerns, API dependence.
• In practice: Most SaaS and apps use this path.

Path B: Open-Source Models + Local Serving. Using open models like LLaMA, Mistral, and Qwen on your own servers.

• Pros: Data privacy, cost control (for large-scale use), customization.
• Cons: Requires infrastructure, performance may not match top commercial offerings, more complex to manage.
• In practice: Internal corporate systems, regulated industries, research labs.

Path C: Hugging Face Transformers. Directly loading, training, and inferencing open models using the PyTorch API.

• Pros: Maximum flexibility, allows for fine-tuning and analysis.
• Cons: Requires optimization for serving, requires GPUs.
• In practice: Research, prototyping, development of specialized models.

Relationship between the three paths: These paths are not mutually exclusive. In practice, you can combine all three. For example: start prototyping with the OpenAI API -> fine-tune a specialized model with Hugging Face -> deploy to production using vLLM for local serving.

The Hugging Face Ecosystem

Hugging Face is at the center of the AI open-source community. It started as a French startup in 2016 and is now essentially the standard for open AI infrastructure.

Model Hub

huggingface.co/models contains hundreds of thousands of pre-trained models. It covers a wide range of domains, including LLMs, vision, audio, and biology.

Notable bio-related models:

• facebook/esm2_t33_650M_UR50D: ESM-2 protein language model (mentioned in Parts 4 and 6).
• microsoft/biogpt-large: Biomedical LLM.
• allenai/scibert_scivocab_uncased: Scientific paper BERT.
• dmis-lab/biobert-v1.1: Pre-trained on PubMed.
• InstaDeepAI/nucleotide-transformer-500m-human-ref: Genomic transformer.

Mainstream LLMs:

• meta-llama/Llama-3.1-70B-Instruct: Meta's large model.
• Qwen/Qwen2.5-72B-Instruct: Alibaba, multilingual and powerful.
• mistralai/Mistral-Large-Instruct-2411: Mistral family.

Transformers Library

This is the standard library for loading and using models from the Hub with PyTorch/JAX/TF.

from transformers import AutoModel, AutoTokenizer

# Load model and tokenizer (automatically downloads)
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Example protein sequence
sequence = "MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"
inputs = tokenizer(sequence, return_tensors="pt")
outputs = model(**inputs)

# Embeddings for each amino acid position
embeddings = outputs.last_hidden_state  # shape: (1, len+2, hidden_dim)

AutoClass: AutoModel, AutoTokenizer, AutoConfig, etc. If you only know the model name, it automatically infers the architecture.

Pipeline: A shortcut for common tasks (summarization, classification, QA, generation).

from transformers import pipeline

qa = pipeline("question-answering", model="deepset/roberta-base-squad2")
result = qa(question="What is TP53?", context="TP53 is a tumor suppressor gene...")
print(result["answer"])

Datasets Library

The standard loader for training and evaluation datasets.

from datasets import load_dataset

ds = load_dataset("nlphuji/mscoco_2014_5k_test_image_text_retrieval")

Accelerate

A wrapper that hides the complexity of multi-GPU, mixed precision, and FSDP.

from accelerate import Accelerator

accelerator = Accelerator(mixed_precision="bf16")
model, optimizer, train_loader = accelerator.prepare(model, optimizer, train_loader)

for batch in train_loader:
    loss = model(**batch).loss
    accelerator.backward(loss)
    optimizer.step()

Hides the DDP and FSDP setup from Part 12.

PEFT: Parameter-Efficient Fine-Tuning

Fine-tuning large models requires hundreds of GB of GPU memory. PEFT (Parameter-Efficient Fine-Tuning) only trains a small adapter.

LoRA (Low-Rank Adaptation): Adds a low-rank (rank 4-16) adapter to large weight matrices. The original parameters are frozen.

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, config)
# Training parameters are less than 1% of the original model

Effect: Allows fine-tuning a 70B model on a single 24GB GPU. Easy to deploy because you only need to store the adapter.

Practical bio example: Fine-tune Llama-3 on a biomedical corpus using LoRA -> create a domain-specific assistant. Only need to manage the adapter file (100-500 MB).

Commercial APIs: Practical Usage

OpenAI

from openai import OpenAI

client = OpenAI()  # OPENAI_API_KEY environment variable

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a molecular biology assistant."},
        {"role": "user", "content": "Summarize TP53 function."}
    ],
    temperature=0.3
)
print(response.choices[0].message.content)

Function Calling (Part 9).

tools = [{"type": "function", "function": {...}}]
response = client.chat.completions.create(
    model="gpt-4o", messages=messages, tools=tools
)

Structured Outputs (Part 7).

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    response_format={"type": "json_schema", "json_schema": {...}}
)

Key models:

• GPT-4o: Multimodal, standard for agents.
• GPT-4o mini: Cheaper, faster. For large-scale tasks.
• o1, o3: Inference-optimized models. For difficult logic and math.

Anthropic Claude

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=1024,
    system="You are a molecular biology assistant.",
    messages=[{"role": "user", "content": "Summarize TP53 function."}]
)
print(response.content[0].text)

Supports Tool Use, Prompt Caching, Vision, Computer Use, etc. Most of the features discussed in Parts 7-10.

Key models:

• Claude Opus 4.7, 4.8: Highest performance. For complex tasks.
• Claude Sonnet 5: Balanced. Suitable for most practical applications.
• Claude Haiku 4.5: Fast and cheap. For large-scale processing.

Google Gemini

import google.generativeai as genai

genai.configure(api_key="...")
model = genai.GenerativeModel("gemini-1.5-pro")
response = model.generate_content("Summarize TP53 function.")
print(response.text)

Gemini Features: 1M+ token context, powerful multimodal capabilities (including video), and affordable pricing.

Comparison of the Three APIs

Practical Selection: There is no right answer. It depends on your team's familiarity, the performance of specific tasks, and price.

Local Model Serving

Local serving is used for internal corporate needs, privacy requirements, and high-volume use cases.

Ollama — Personal/Small Scale

Easily run open-source models on your local laptop. Supports Mac, Linux, and Windows.

# After installation
ollama pull llama3.1
ollama run llama3.1

Also provides an API.

import requests

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.1",
    "prompt": "Summarize TP53 function.",
    "stream": False
})
print(response.json()["response"])

Ollama Features: Supports GGUF quantized models (4-bit, 5-bit, 8-bit). Optimized for Mac Apple Silicon. A standard for local experimentation by developers and researchers.

Capacity: Llama-3.1-8B Q4 requires 5GB and can run on a 16GB RAM Mac. 70B Q4 requires 40GB. Possible on a Mac with M3 Max and 128GB.

vLLM — Production Serving

Production-scale LLM serving. Maximizes throughput through continuous batching and PagedAttention.

pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4

Provides an OpenAI-compatible API. Can be called using existing OpenAI clients.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[...]
)

vLLM Features: PagedAttention (memory-efficient KV cache), continuous batching (maximizes GPU utilization). Throughput is 10-24x higher than pure Hugging Face.

TGI, SGLang, LMDeploy

Alternatives to vLLM. Each has different strengths.

• TGI (Hugging Face): Integrates with the Hugging Face ecosystem, easy Docker deployment.
• SGLang: Optimized for complex structured prompts (RAG, agents).
• LMDeploy: Strong in the Chinese community. Optimized for Qwen and InternLM.

Llama.cpp

A lightweight inference engine based on C++. Runs GGUF models. Used internally by Ollama. Supports CPU, CUDA, and Metal (Mac). Suitable for personal development and edge deployment.

Embedding API — Recycle from Part #8

The embeddings needed for RAG in Part #8 can be obtained through both API and local methods.

API

# OpenAI
response = openai_client.embeddings.create(
    model="text-embedding-3-large",
    input="protein sequence description"
)
embedding = response.data[0].embedding  # 3072-dim

# Cohere
co_response = cohere_client.embed(texts=["..."], model="embed-english-v3.0")

# Voyage AI (Top performer in benchmarks)
voyage_response = voyage_client.embed(texts=["..."], model="voyage-3")

Hugging Face Local

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-en-v1.5")
embeddings = model.encode(["chunk 1", "chunk 2"])

Bio-specific:

• PubMedBERT sentence transformer: Fine-tuned for biomedical tasks.
• BioBERT-based: Trained on PubMed and MEDLINE.
• ProtT5 and ESM sentence embeddings: Specialized for protein sequences.

Fine-Tuning Options

In Part #8, we compared fine-tuning versus RAG. These are the options for fine-tuning itself.

Full Fine-Tuning

Updates all parameters. Accurate but expensive.

• When: Large amounts of domain data, fundamental style changes.
• Cost: 70B model = millions to tens of millions of won in GPU time.

LoRA/QLoRA

Trains only a few adapter layers.

• When: Most cases. Small data, low cost.
• Cost: 70B LoRA = a few hours on one A100.

Full-Precision Fine-Tuning API

OpenAI and Anthropic provide managed fine-tuning APIs.

# OpenAI example
job = openai_client.fine_tuning.jobs.create(
    training_file="file-abc",
    model="gpt-4o-mini-2024-07-18"
)

• When: Customizing commercial models.
• Cost: Depends on data size and number of epochs. Most cases, a few tens of thousands of won.

Data Requirements

• LoRA: 500-5000 examples are sufficient for significant improvement.
• Full fine-tuning: 10,000+ examples.
• Continual pre-training: (large amounts of domain text): hundreds of millions to billions of tokens.

Practical Workflow

Typical project flow:

1. Prototype: OpenAI/Anthropic API + prompt engineering. A few days to 1 week.
2. RAG Implementation: Domain data vector database. Part #8. 1-2 weeks.
3. Agentification: Tools and loops. Part #9. 2-4 weeks.
4. Performance Improvement: Profile to determine the bottleneck.
   • If the problem is the prompt: Use a better system prompt.
   • If a specific style is needed: LoRA fine-tuning.
   • If privacy is a concern: Migrate to a local open model.
   • If cost is a concern: Use self-serving.
5. Production Deployment: Monitoring, error handling, rate limiting, audit logs.

Example Bio Lab Project: Develop a paper summarization assistant.

• Week 1-2: Prototype with OpenAI GPT-4o + PubMed RAG.
• Week 3-4: Collect feedback from conference presentations.
• Week 5-8: Improve domain performance with BioBERT embeddings + migrate to Claude Sonnet.
• Month 3-4: Deploy vLLM + Llama-3 on a local lab server. Data privacy.
• Month 6+: Continuous fine-tuning and reinforcement of citation verification layer.

Bio Application Scenarios

Scenario 1: Protein Sequence Embedding Pipeline

Apply RAG from Part #8 to protein sequence search.

from transformers import AutoTokenizer, AutoModel
import torch

model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name).cuda().eval()

def get_protein_embedding(sequence):
    inputs = tokenizer(sequence, return_tensors="pt").to("cuda")
    with torch.no_grad():
        outputs = model(**inputs)
    # Mean pooling (average over amino acid positions)
    return outputs.last_hidden_state.mean(dim=1).cpu().numpy()

# Embed multiple proteins
proteins = ["MKTVRQ...", "MSPTQR...", ...]
embeddings = np.stack([get_protein_embedding(p) for p in proteins])

# Store in a vector database -> search for similar proteins

Used for predicting enzyme activity and protein-protein interactions.

Scenario 2: Domain-Specific Chatbot (LoRA + vLLM)

Fine-tune Llama-3-8B with 500 lab papers using LoRA, and serve with vLLM.

• Week 1: 500 papers -> create JSON-format QA pairs (automatically generated with GPT-4o).
• Week 2: Train LoRA with Hugging Face + PEFT.
• Week 3: vLLM serving + Slack bot integration.

Assistant optimized for internal lab knowledge and recent papers. Good for privacy and cost.

Scenario 3: Multimodal Pathology Image Analysis

Use a vision-language model (e.g., LLaVA) to analyze pathology slide images and text.

• Open source: llava-hf/llava-v1.6-mistral-7b-hf
• Commercial: GPT-4o Vision, Claude 3.5 Vision, Gemini 1.5 Pro

Identify cell types in images and generate natural language descriptions. A practical application of Scenario 3 from Part #6.

Key Takeaways

• Three paths for model usage: commercial API, open-source model local serving, direct Hugging Face. Not mutually exclusive, can be combined.
• Hugging Face Hub is the standard for open-source models and datasets. Transformers, Datasets, Accelerate, PEFT libraries.
• The three major commercial APIs: OpenAI (GPT), Anthropic (Claude), and Google (Gemini). They differ in context, price, and specialization.
• Local serving: Ollama (personal), vLLM (production), TGI, SGLang, llama.cpp.
• Use LoRA/QLoRA for low-cost fine-tuning of large models. Only manage the 100-500MB adapter layers.
• Practical workflow: Prototype (API) -> RAG -> Agent -> Performance Improvement -> Production Deployment.
• Bio-specific models: ESM, BioGPT, SciBERT, BioBERT, Nucleotide Transformer, etc. Available for immediate use on Hugging Face.

📐 Appendix — Practical Optimization for Experts

Difficulty: Very Hard Target Audience: Readers with practical experience in serving and fine-tuning production LLMs

A.1 LoRA Equations

Original weight matrix W ∈ ℝ^{d × d}. LoRA:

W_effective = W + ΔW = W + B · A

• A ∈ ℝ^{r × d} (low-rank down projection)
• B ∈ ℝ^{d × r} (low-rank up projection)
• r ≪ d (e.g., r=16, d=4096)

Trainable Parameters: Only A and B. 2·r·d parameters. Compared to the original d², this is 2r/d ≈ 0.008. Significantly fewer.

Initialization: A ~ Normal distribution, B = 0. At the beginning of training, ΔW = 0.

Scaling:

W_effective = W + (α/r) · B · A

α is a hyperparameter that ensures a consistent learning signal regardless of the rank.

A.2 QLoRA — 4-bit Base + LoRA

The original model is quantized and stored in 4-bit. The LoRA adapter is FP16. During training, only the necessary parts of the original weights are dequantized.

• NF4 (NormalFloat4): 4-bit float, optimized for quantization with a normal distribution.
• Double Quantization: The quantization constants themselves are re-quantized, providing further savings.
• Paged Optimizer: Optimizer states are offloaded to CPU memory.

Effect: Allows fine-tuning a 70B model on a single 24GB GPU. This paper (Dettmers et al., 2023) has become the standard for fine-tuning open models.

A.3 vLLM PagedAttention

Addresses the KV cache problem from Episode #6.

Traditional Approach: Each request reserves continuous memory equal to the maximum context size it requested. Even if only a portion is used, the reserved size cannot be reduced, leading to significant memory waste.

PagedAttention: The KV cache is divided into pages (e.g., 16 tokens) and allocated only when needed. This is similar to the virtual memory system in operating systems.

Effect:

• Memory utilization increases from 60% to 96%+
• When combined with continuous batching, throughput increases by 10-24x.

A.4 Continuous Batching

Traditional Approach: Wait for the batch to complete; all requests in the batch move to the same step.

Continuous: Each request is replaced with a new request as soon as it is completed. Minimizes GPU idle time.

Time Loss: The prefill step (processing the prompt) only occurs once at the beginning of the request. Only the decode step is continuous.

Recent Developments: Chunked Prefill (divides long prompts into chunks for processing), Speculative Decoding (a small draft model generates candidate outputs, which are then verified by the large model). Reduces latency by 30-50%.

A.5 Quantization

Stores parameters and activations in lower bit widths.

Post-Training Quantization (PTQ): Quantization performed after training. Fast.

• GPTQ: Group-wise minimum error quantization. A 4-bit standard.
• AWQ (Activation-aware Weight Quantization): Considers the activation distribution. Replaces GPTQ.

Quantization-Aware Training (QAT): Simulates quantization during training. More accurate, but more computationally expensive.

FP8: Supported by H100 generation. Minimal accuracy loss, 2x speed and memory improvement.

A.6 Serving Benchmark Metrics

Throughput (Requests/second): The number of requests processed per second.

TTFT (Time To First Token): The latency from the request to the first token. Affects user experience.

ITL (Inter-Token Latency): The latency between tokens. Affects streaming speed.

Concurrent Users: The number of users processed concurrently.

Measurement Tools: vllm/benchmarks, llmperf, guidance-serve.

A.7 RLHF Fine-tuning in Practice

TRL Library (HuggingFace): Implements RLHF and DPO.

from trl import DPOTrainer, DPOConfig

config = DPOConfig(
    beta=0.1, learning_rate=5e-7,
    per_device_train_batch_size=4, num_train_epochs=1
)
trainer = DPOTrainer(
    model=model, ref_model=ref_model,
    args=config, train_dataset=preference_dataset,
    tokenizer=tokenizer
)
trainer.train()

The Entire RLHF Pipeline: SFT → Reward Model → PPO. Each step is a project in itself. DPO compresses this into a single step.

A.8 Instruction Tuning Data Format

Alpaca Format:

{"instruction": "...", "input": "...", "output": "..."}

ShareGPT Format (conversation):

{"conversations": [
    {"from": "human", "value": "..."},
    {"from": "gpt", "value": "..."}
]}

Chat Template: Different formats for each model.

tokenizer.apply_chat_template(messages, tokenize=False)
# "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n..."

A.9 Data Refinement Pipeline

Fine-tuning performance depends on data quality.

Duplicate Removal: Remove similar examples using MinHash and SimHash. Quality Filtering: Use a perplexity threshold or rule-based rules. Diversity Assurance: Use cluster-based sampling. Reduce Label Noise: Use human review or multiple labelers.

Constitutional AI Format Self-Improvement: Train on data where the LLM criticizes and improves its own answers. Episode #11.

A.10 Production Monitoring

LangSmith, Weave, Braintrust: LLM observability tools.

Logs: Prompts, responses, time, cost, errors. Evaluation: Accuracy, groundedness, safety. Use LLM-as-a-judge for automation. Drift Detection: Alert on changes in response distribution. A/B Testing: Compare prompts and model versions.

Essential infrastructure for production LLM systems.

References

All content, scenarios, analogies, and figures in this episode are developed by BioPlayground, and the following are external references that may be helpful for learning the concepts.

• HuggingFace Official: huggingface.co
• Transformers Documentation: huggingface.co/docs/transformers
• PEFT Documentation: huggingface.co/docs/peft
• LoRA Paper: Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (ICLR 2022)
• QLoRA: Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" (NeurIPS 2023)
• vLLM Paper: Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023)
• Ollama: ollama.ai
• OpenAI API: platform.openai.com/docs
• Anthropic Claude API: docs.anthropic.com
• Google Gemini API: ai.google.dev
• TRL (RLHF): huggingface.co/docs/trl
• ESM: github.com/facebookresearch/esm
• BioGPT: github.com/microsoft/BioGPT

Episode #13 completes the map of today's model ecosystem. Episode #14 will cover practical applications of AI coding agents.

Next Concept

• Ep. #14 claude-code-and-cursor — AI Coding Agent in Action. Comparing Claude Code, Cursor, and Windsurf.
• Ep. #15 bio-ai-integration — Phase 4. Integrating Everything So Far into a Bio Pipeline.