ECE-GY 6123 – Image and Video Processing (Fall 2024)

Course Description: This course introduces fundamentals of image and video processing, including color image capture and representation; contrast enhancement; spatial domain filtering; two-dimensional (2D) Fourier transform and frequency domain interpretation of linear convolution; image sampling and resizing; multi-resolution image representation using pyramid and wavelet transforms; feature point detection and feature correspondence; geometric transformation, image registration, and image stitching; video motion characterization and estimation; video stabilization and panoramic view generation; image representation using orthogonal transforms; sparsity-based image recovery; basic image compression techniques and standards (JPEG and JPEG2000 standard); video compression using adaptive spatial and temporal prediction; video coding standards (MPEGx/H26x); Stereo and multi-view image and video processing (depth from disparity, disparity estimation, video synthesis, compression). Basics of deep learning for image processing and computer vision will also be introduced. Students will learn to implement selected algorithms in Python. Prior experience with Python and deep learning are not required. You will learn as the course progresses. A class project, preferably in teams of 2 to 3 people, is required.

Prerequisites: Graduate status. ECE-GY 6113 and ECE-GY 6303 preferred but not required. Should have good background in linear algebra / matrix theory, probability, and signals and systems. Undergraduate students must have completed EE-UY 3054 Signals and systems and EE-UY 2233 Probability, and linear algebra.

Instructor: Professor Yao Wang, 370 Jay Street, Rm 957, (646)-997-3469, Email: yaowang at nyu.edu. Homepage Office hour: Mon. 4:00-5:00 PM (online), Wed. 4:30-6:00 PM (after class or in office). Monday hour Zoom links available on Brightspace. Contact me via email to schedule other times.

Teaching Assistants: Asif Mammadov (am14089 at nyu.edu). Office Hour: Tues 11:00-12:00 and Thurs 11:00-12:00. In person: 370 Jay St, rm 966 (also available over zoom if you cannot come in person). Zoom link available on Brightspace.

Course Schedule: In person: Wed. 2:00 PM – 4:30 PM, JAB 473, Brooklyn.

Text Book/References: Richard Szeliski, Computer Vision: Algorithms and Applications. 2nd Edition (Sept. 30,2021 version) (Available online: Link) (Cover most of the material, except sparsity-based image processing and image and video coding).
(Optional) Y. Wang, J. Ostermann, and Y.Q.Zhang, Video Processing and Communications. Prentice Hall, 2002. Link (Reference for Fourier transforms, image and video coding, motion estimation, and stereo).
(Optional) R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice Hall, (3rd Edition) 2008. ISBN number 9780131687288. Link (Good reference for basic image processing, wavelet transforms and image coding).

Course Structure: The class will consist of weekly lectures, weekly written homework assignments (not graded, solution will be given), roughly biweekly short quizzes (based on homework assignment), computer assignments (CA), a team project (2-3 people in a team). There will be two optional tutorials outside the class time, one to introduce Python programming, another to introduce PyTorch and Google Cloud Platform.

Grading: Quizzes: 40%, Computer assignments: 30%, Project: 30%. Project grade depends on project proposal (2%), midterm project report (3%), final report (5%), project presentation (10%), and technical accomplishment (10%).

Attendance: Students are expected to attend all lectures and quizzes in-person.

Homework: Written HW will be assigned after each lecture but not graded, and solutions will be provided. Programming assignments will be due as posted. Each assignment counts for 10 points. Late submission of programming assignments will be accepted up to 3 days late, with 2 pt deduction for each day. Students can work in teams, but you must submit your own solutions. Solutions to computer assignments will be posted 1 week after the due date. We will aim to complete the grading of each quiz and computer assignment within 1-2 weeks.

Quiz: A quiz will be held biweekly. The total time for each quiz is 20 minutes. The quiz problems will be similar to the written HW problems and/or review questions in the lecture note.

Tentative Course Schedule (lecture notes may be updated shortly before the lecture date)

Project Guideline: Link

Suggested Project List: Link (Updated 9/16/2023)

Sample Data: Sample Images Middlebury Stereo Image Database

Policy on Academic Integrity: The School of Engineering encourages academic excellence in an environment that promotes honesty, integrity, and fairness. Please see the policy on academic dishonesty: Link to NYU Tandon Policy, Link to NYU Policy.

Inclusion Statement: The NYU Tandon School values an inclusive and equitable environment for all our students. I hope to foster a sense of community in this class and consider it a place where individuals of all backgrounds, beliefs, ethnicities, national origins, gender identities, sexual orientations, religious and political affiliations, and abilities will be treated with respect. It is my intent that all students’ learning needs be addressed both in and out of class, and that the diversity that students bring to this class be viewed as a resource, strength, and benefit. If this standard is not being upheld, please feel free to speak with me. Please visit this link for NYU Tandon’s effort in diversity and inclusion.

Moses Center Statement of Disability: If you are a student with a disability and would like to request accommodations, please contact New York University’s Moses Center for Students with Disabilities (CSD). You must be registered with CSD to receive accommodations. Information about the Moses Center can be found at www.nyu.edu/csd. The Moses Center is located at 726 Broadway on the 3rd floor.

Links to Resources (lecture notes) in Previous Offerings:

Other Useful Links:

Sample Exams:

Sample Images: