11-767: On-Device Machine Learning

Time: 3:30-4:50 Tuesdays & Thursdays    Place: 7500 Wean Hall

On-Device Machine Learning is a project-based course covering how to construct, evaluate, and deploy models that can run on low-resource hardware (e.g. smartphones and mobile robots). The course is primarily focused on how modern capabilities in e.g. language understanding and computer vision, enabled by deep neural networks, can be achieved on hardware with limited memory and compute bandwidth. Towards this end the course covers advanced approaches such as knowledge distillation, pruning, quantization, and parameter-efficient fine-tuning, as well as task-specific model design and efficiency benchmarking.

Techniques will be taught in lectures, and students will have the opportunity to develop hands-on experience through programming exercises and a course project. For the course project, students will work in teams to deploy a custom machine learning model on a low-power device, demonstrating that they have achieved improved efficiency over a reasonable baseline in their chosen task and hardware setting. Students are encouraged to bring their own hardware and/or data for class projects, but instructors can also provide low-power compute hardware (e.g. SBCs and inference accelerators) in addition to sensors (e.g. microphones, cameras, and robotics) for the course project.¹ The projects will be evaluated under a mAh “budget” equivalent to one full charge of a smart phone battery (~4 Ah or 2h on Jetson Nano, 7h on Raspberry Pi 4, or 26h on a Raspberry Pi Zero W).

Example Industry Motivation: “…if the coffee maker with voice recognition was in use for four years, the speech recognition cost for chewing on data back in the Mr Coffee datacenter would wipe out the entire revenue stream from that coffee maker, but that same function, if implemented on a device specifically tuned for this very precise job, could be done for under $1 and would not affect the purchase price significantly.” – Source.

• Assignments will be submitted via Canvas.
• Announcements, discussion and Q&A will happen on Piazza. Please direct all questions or concerns regarding the course to Piazza; Do not email individual course staff unless instructed to do so. Ensuring that all course staff see your message by posting to Piazza will lead to the shortest response times.

Instructors

Emma Strubell

Assistant Professor, LTI

TAs

Jared Fernandez

PhD student, LTI

Office hours: Wed 3:30-4:30pm, GHC 5417

Vishruth Veerendranath

MIIS student, LTI

Office hours: Thu 2-3pm, GHC 5417

Vibha Masti

MCDS student, LTI

Office hours: Mon 12-1pm, GHC 5417

Schedule

Schedule is subject to change, this will always be the latest version.

Grades are based on a combination of individual and group assignments, including labs, quizzes, lecture highlights, and deliverables for the semester-long class project. All assignments are due at 11:59pm ET on the specified date.

• Lecture highlights (18 points). (11 assignments; 2 points each. Two Lowest grades dropped. Individual.) Lecture highlights are meant to be lightweight assignments to help encourage engagement in and reflection on class lectures by writing a few sentences describing key takeaways from the lecture. Not all lectures have an associated highlight.

• Labs (32 points). (4 assignments; 8 points each. First lab is individual, the rest are group.) Labs are coding-heavy assignments in Python/PyTorch designed to give hands-on experience implementing the methodologies discussed in class. After the first lab, labs will be group assignments to be completed with project teams using the codebase being developed for your course project.

• Quizzes (15 points). (4 in-class quizzes; 5 points each. Lowest grade dropped. Individual.) The beginning of “project hours” classes will be devoted to an in-class quiz. Quizzes will cover topics covered in previous lectures, but will not be cumulative (i.e. quizzes will cover only topics discussed since the last quiz.)

• Project (34 points). (Group.) A semester-long 3-4 person team project focused on deploying a machine learning model on a low-compute device (SBC). More details above and in class. There will be intermediate assignments (project proposal, project sharing) as well as a final presentation and report.

Communication. Important information about the course such as changes to the syllabus, due dates, office hours, etc. will be communicated in class as well as on Piazza. Grades will be available via Canvas, and slides will be posted on Canvas. Students are encouraged to post questions (and answers!) on Piazza, using anonymous/limited audience posting as appropriate. Everyone, both students and course staff, is expected to check Piazza regularly, and to communicate clearly, kindly, and with respect.

Attendance. Students attending CMU in Pittsburgh are expected to attend every class in person. That being said, we understand that scheduling conflicts may arise infrequently throughout the semester, such as a job interview, conference, or illness. Lectures will be recorded and access to missed lectures can be requested using the form provided on Canvas/Piazza.

Late work. In addition to dropping the lowest highlight and quiz grades, each student and each group is allocated five total late days (calculated separately per-group and per-student) that may be used as needed throughout the semester. They can be used all at once, one-at-a-time, or anything in between. If a student hands in a homework assignment two days late, this will use two of their late days (leaving three days if no late days had been used previously). Lateness is rounded up to the nearest day. Group late days may be used for the final project report.

There is no grade penalty for using late days. However, after late days are exhausted, late work will not be accepted for credit. The only exceptions to this are unexpected events—the death of a friend or family member, a severe illness, a mental health crisis or episode, etc. These exceptions will be handled on a case-by-case basis with the instructors. Students are encouraged to reach out to instructors as soon as they anticipate requiring accommodations beyond the grace day limit, ideally in advance of the assignment due date. Because of this policy, students are urged to manage their late days with care and to save them for dire times.

Academic honesty. It will be clearly indicated which assignments are to be completed individually or as groups in this class. Verbal collaboration on individual assignments is acceptable, but everything you turn in must be your own work, and you must note the names of anyone you collaborated with on the assignment and cite resources that you used. This includes LLM chatbots like ChatGPT used in any way to assist in completing assignments. The project is to be completed by a team, with every member contributing equally. You are encouraged to use existing software and frameworks in your project; you must acknowledge these appropriately in the documentation and project report. Project assignments will include a section listing contributions of each team member, and discrepancies between what is reported and here and actual allocation of work is considered an academic integrity violation. Suspected violations of academic integrity rules will be handled in accordance with the CMU guidelines on collaboration and cheating.

Accommodations for students with disabilities. If you have a disability and have an accommodations letter from the Disability Resources office, we encourage you to discuss your accommodations and needs with the instructors as early in the semester as possible. We will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, we encourage you to contact them at access@andrew.cmu.edu.

Take care of yourself! As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, feeling down, difficulty concentrating and/or lack of motivation. This is normal, and all of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of a healthy life is learning how to ask for help. Asking for support sooner rather than later is almost always helpful. CMU services are available free to students, and treatment does work. You can learn more about confidential mental health services available on campus through Counseling and Psychological Services (CaPS). Support is always available (24/7) at: 412-268-2922.

Take care of your classmates and instructors! In this class, every individual will and must be treated with respect. The ways we are diverse are many and are fundamental to building and maintaining an equitable and inclusive campus community. These include but are not limited to: race, color, national origin, caste, sex, disability (visible or invisible), age, sexual identity, gender identity, religion, creed, ancestry, belief, veteran status, or genetic information.

Research shows that greater diversity across individuals leads to greater creativity in the group. We at CMU work to promote diversity, equity and inclusion not only because it is necessary for excellence and innovation, but because it is just. Therefore, while we are imperfect, we ask you all to fully commit to work, both inside and outside of our classrooms to increase our commitment to build and sustain a campus community that embraces these core values. It is the responsibility of each of us to create a safer and more inclusive environment. Incidents of bias or discrimination, whether intentional or unintentional in their occurrence, contribute to creating an unwelcoming environment for individuals and groups at the university. If you feel that you, or those around you, have been treated unfairly based upon their identity (or perceived identity) by us, by other members of the teaching staff, or by other students in the course, we ask that you share your experience with the Center for Student Diversity and Inclusion (csdi@andrew.cmu.edu, (412) 268-2150) or by using CMU’s anonymous reporting platform.