Course Detail
Foundations and Applications of Large Language Models
This course examines the theoretical foundations, system design, and real-world uses of large language models. It emphasizes both principled understanding and research-driven practice, from pretraining and adaptation to evaluation, safety, and deployment.
Lecture Schedule
Why Large Language Models Matter
The evolution from statistical NLP to pretrained language models, the emergence of scaling laws, and the current research landscape.
References: Brown et al. (2020); Kaplan et al. (2020).
Transformer Foundations
Self-attention, positional encoding, feed-forward blocks, training objectives, and why transformers became the dominant architecture.
References: Vaswani et al. (2017); The Illustrated Transformer.
Pretraining Objectives and Data Pipelines
Causal language modeling, masked language modeling, tokenization, web-scale corpora, and data quality considerations.
References: Devlin et al. (2019); Raffel et al. (2020); Penedo et al. (2023).
Scaling Laws and Emergent Ability
How model size, compute, and data interact; what emergence means; and the debate around capability forecasting.
References: Kaplan et al. (2020); Hoffmann et al. (2022); Wei et al. (2022).
Instruction Tuning and Alignment
Supervised instruction tuning, preference learning, RLHF, DPO, and the role of alignment data in model behavior.
References: Ouyang et al. (2022); Rafailov et al. (2023).
Parameter-Efficient Fine-Tuning
Adapters, prefix tuning, LoRA, QLoRA, and practical strategies for adapting LLMs under resource constraints.
References: Houlsby et al. (2019); Li and Liang (2021); Hu et al. (2022); Dettmers et al. (2023).
Prompting and In-Context Learning
Zero-shot and few-shot prompting, chain-of-thought reasoning, self-consistency, and prompt design patterns.
References: Brown et al. (2020); Wei et al. (2022); Wang et al. (2023).
Retrieval-Augmented Generation
Knowledge grounding, dense retrieval, indexing, retrieval orchestration, and hallucination reduction.
References: Lewis et al. (2020); Gao et al. (2024 survey).
Evaluation of LLMs
Automatic metrics, benchmark design, task contamination, human evaluation, and robustness considerations.
References: Liang et al. (2022); Chang et al. (2024 survey).
Reasoning and Tool Use
Program-aided reasoning, tool calling, agent workflows, planning, and the limits of purely autoregressive reasoning.
References: Schick et al. (2023); Yao et al. (2023); Qin et al. (2023).
Efficiency and Deployment
Quantization, KV-cache optimization, distillation, serving stacks, and latency-quality trade-offs.
References: Dettmers et al. (2022); Xiao et al. (2023); relevant systems papers.
Safety, Security, and Privacy
Jailbreaks, prompt injection, unsafe generation, privacy leakage, and defense strategies for safe deployment.
References: Ganguli et al. (2022); Zou et al. (2023); OWASP LLM guidance.
Multilingual and Domain-Specific LLMs
Cross-lingual transfer, data scarcity, vertical-domain modeling, and adaptation for specialized applications.
References: mT5; BLOOM; selected domain adaptation papers.
Multimodal Large Models
Vision-language alignment, multimodal instruction tuning, and interfaces that combine text with images and other modalities.
References: CLIP; Flamingo; LLaVA; GPT-4V report.
Research Paper Clinic
Students present recent papers, compare methods, critique evaluations, and identify promising research directions.
References: Selected ACL, EMNLP, ICLR, ICML, and NeurIPS papers.
Open Problems and Final Project Discussion
Reasoning reliability, data efficiency, controllability, interpretability, agentic systems, and future research opportunities.
References: Recent survey papers and frontier research articles.
Core References
- Vaswani et al. (2017). Attention Is All You Need.
- Brown et al. (2020). Language Models are Few-Shot Learners.
- Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback.
- Raffel et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
- Recent surveys on RAG, evaluation, alignment, and efficient fine-tuning.
Assignments and Assessment
- Reading Reports: Short reports on representative LLM papers and surveys.
- Paper Presentation: Each student or group presents one recent research paper and leads discussion.
- Mini Project: A small empirical project on prompting, fine-tuning, retrieval augmentation, evaluation, or safety.
- Final Assessment: Project report, presentation, or research proposal.
Lecture Slides and Course Materials
- Week 1-4 lecture slides: to be uploaded
- Week 5-8 lecture slides: to be uploaded
- Week 9-12 lecture slides: to be uploaded
- Week 13-16 lecture slides: to be uploaded
- Paper list and project guidelines: to be uploaded
You can later replace these placeholders with direct PDF, PPT, notebook, or GitHub resource links.