Small Language Models in Action

While large language models have transformed generative AI (GenAI), their heavy compute demands, high inference costs, and cloud dependency make them impractical for many real-world applications. Small language models (SLMs) offer a compelling alternative, delivering powerful GenAI capabilities while running efficiently on mobile devices, edge hardware, and cost-constrained systems. This book bridges the gap between theoretical model architecture and real-world deployment, serving as your ultimate engineering roadmap for building efficient, compact, and production-ready AI systems.

This book systematically guides you through the entire life cycle of small language models across 14 comprehensive chapters. This book begins with the foundations of SLMs, covering their architecture, design principles, and size-performance trade-offs. Readers will master core compression techniques including knowledge distillation, quantization, and pruning to shrink models while preserving accuracy. The book also explores lightweight transformers, efficient fine-tuning, and inference optimization for latency and memory, followed by practical deployment on edge and mobile platforms using ONNX and TensorFlow Lite. Finally, the book explores SLM-powered agents capable of tool use and multi-step reasoning, hybrid architectures that combine small and large models, and real-world case studies, closing with benchmarking and cost-performance frameworks.

By the end of this book, readers will be able to build production-ready SLMs from scratch, apply compression techniques confidently, and deploy efficient generative AI across mobile, edge, and cloud, making sound architectural decisions under real-world constraints.

WHAT YOU WILL LEARN
● Design compact SLM architectures balancing size and performance.
● Evaluate SLMs across multiple dimensions.
● Apply distillation, quantization, and pruning to compress models.
● Fine-tune small models for task and domain-specific generative AI tasks.
● Optimize inference for low latency and minimal memory.
● Deploy SLMs on edge, mobile, and cloud platforms.
● Build SLM-powered agents with tool use and reasoning.

WHO THIS BOOK IS FOR
The book is designed for students, machine learning engineers, NLP practitioners, software developers, and AI builders deploying efficient models to resource-constrained environments, edge systems, and mobile devices. Readers need working knowledge of Python alongside a basic understanding of core machine learning concepts.

1. Small Language Model Architecture Overview
2. Transformer Architecture Fundamentals for SLMs
3. Evaluating SLMs
4. Knowledge Distillation for Language Models
5. Quantization for Efficient Inference
6. Pruning and Sparsity for Compact Models
7. Fine-tuning SLMs
8. Optimizing Inference for Speed and Memory
9. Edge and Mobile Deployment for Generative AI
10. Production Pipelines and Serving Infrastructure
11. SLMs in Agents and Agentic Workflows
12. Hybrid Architectures Combining SLMs and LLMs
13. Multimodal SLMs
14. SLM Production Case Studies and Trends

Dr. Sireesha Muppala is a visionary technology leader with over 25 years of experience driving digital transformation and unlocking the potential of cutting-edge innovations. As a senior solutions architecture leader at Amazon Web Services (AWS), Sireesha spearheads efforts to empower businesses with the transformative power of generative AI and other emerging technologies. She currently leads a team of dedicated technologists at AWS, focused on partnering with leading organizations in the automotive and manufacturing sectors to unlock the full potential of generative AI and other cutting-edge cloud-based technologies.


Sireesha's remarkable career has been fueled by her passion for technology and her keen ability to identify disruptive trends. Sireesha received her Ph.D. in computer science from the University of Colorado, Colorado Springs. Prior to joining AWS, she held leadership roles at organizations of all sizes across diverse industries, including Oracle, Blackhawk Network, Primer AI where she was instrumental in developing and implementing groundbreaking solutions that addressed complex business challenges.


Sireesha's technical expertise is complemented by her strategic vision and collaborative approach, which have earned her a reputation as a trusted advisor to C-suite executives and industry leaders. An accomplished author and sought-after speaker, Sireesha has authored numerous research papers, white papers, and books she frequently shares her insights on the transformative power of emerging technologies at industry conferences and events, inspiring audiences with her deep understanding of the evolving technology landscape and its impact on the future of business.


Beyond her professional achievements, Sireesha is a passionate advocate for STEM education and empowering underrepresented communities in the technology sector. She is a co-founder of the Denver chapter of Women in Big Data, where she leverages her expertise to create meaningful change and inspire the next generation of innovators.


With her unwavering commitment to innovation and her ability to translate complex technical concepts into practical business solutions, Sireesha Muppala is poised to continue leading the charge in the ever-evolving world of emerging technologies at AWS, driving transformative change and unlocking new frontiers of growth for organizations.



Randy DeFauw is a technologist at heart. While majoring in electrical engineering at the University of Michigan, he took part in an autonomous vehicle research program. That started his journey into signal processing and what we now call computer vision. Randy applied his image processing expertise on a major defense program in San Diego, before deciding to pursue more customer-facing roles.


Randy worked in consulting, product management, and architecture for several years. In the course of that journey, he worked at a startup that was making distributed storage technology for big data systems. That role reignited his passion for working with data, and also gave him a solid grounding in modern distributed system design. At this point he picked up an MBA to give him the business grounding to go along with his two electrical engineering degrees. He then led a private cloud architecture team on a multi-year project. The intersection of data and cloud was the ideal spot for him, as the cloud unlocked so much potential for data and ML Projects.


Randy then moved to AWS, where he has helped customers architect a variety of cloud applications. He focuses on the data ecosystem, from big data to machine learning and now generative AI. Randy likes to focus on the unsolved problem areas, like LLM interpretation and solving operations research problems through reinforcement learning. He publishes frequently in various channels, and often authors sample code published on GitHub. He is a trusted technical advisor to customers and still a hands-on practitioner.


Since his days as a teaching assistant in university, Randy never lost his enthusiasm for helping others learn. He works closely with early career colleagues, mentors at the high school and college level, and is an advisor to a Colorado K-12 education program on STEM initiatives.


Shelbee Eigenbrode is passionate about innovation and using data along with AI to drive business outcomes. As a worldwide genai/ml specialist solutions architect and data scientist leader with over 20 years of experience in AI/ML and cloud solutions. Her work centers on the advanced applications of AI and GenAI at scale including customization of open-source foundation models, optimizing inference for cost and latency at scale, and operationalizing agentic and GenAI workloads in production.

Shelbee began her career as a software engineer at IBM, building and delivering solutions across a range of products and technologies spanning database administration, cloud infrastructure, analytics platforms, and AI product solutions. That engineering foundation carried into a move to AWS in a customer-facing role, where she has spent her career since helping customers apply AI solutions to drive real business outcomes. Most recently, she has led a global organization of specialist solutions architects and data scientists focused on advanced model customization, advanced inference, and agentic workflows, following prior roles as principal AI/ML specialist solutions architect.

Shelbee holds more than 35 patents, is a three time published author and also co-created the Coursera course, which has trained over 400,000 learners. She is a frequent speaker and recognized thought leader on AI.

Beyond her technical work, Shelbee is a co-founder of the Denver chapter of Women in Big Data, where she leverages her expertise to create meaningful change and inspire the next generation of innovators.