Advanced technique in data analysis and statistical learning

This course provides a comprehensive introduction to Machine Learning, covering both theoretical foundations and practical applications. It introduces students to supervised and unsupervised learning, focusing on model evaluation, feature selection, clustering, decision trees, and regression techniques.
Students will develop hands-on expertise in building, tuning, and evaluating models while understanding the mathematical principles behind them. The course emphasizes best practices in data preprocessing, model selection, and performance optimization, equipping students with the necessary tools to apply Machine Learning in real-world scenarios across various industries.

Linear & Logistic Regression

• Simple vs. Multiple Linear Regression
• Assumptions (Gauss-Markov Theorem)
• Model Inference & Statistical Testing (t-tests, F-tests, Confidence Intervals)
• Logistic Regression for Classification
• Handling Categorical Variables (Dummy Encoding, Polynomial Regression)
• Practical Example: Predicting house prices with multiple regression

Decision Trees

• Tree-Based Model Structure
• Splitting Criteria (Gini Index, Entropy)
• Overfitting Prevention (Pre-Pruning, Post-Pruning)
• Computational Complexity Considerations
• Practical Example: Decision tree classification on real-world data

Model Evaluation & Feature Selection

• Understanding Bias-Variance Trade-off
• Cross-Validation techniques
• Evaluation Metrics (MAE, RMSE, R², AUC-ROC, Confusion Matrix)
• Feature Selection Techniques (Wrapper, Filter, Regularization)
• Hyperparameter Tuning (Grid Search, Bayesian Optimization, AutoML)
• Practical Example: Model selection and tuning using Python

Clustering Techniques

• Introduction to Unsupervised Learning
• Hierarchical Clustering (Agglomerative, Divisive)
• Partitioning Methods (K-Means, K-Medoids)
• Density-Based Clustering (DBSCAN, OPTICS)
• Gaussian Mixture Models with Expectation-Maximization
• Practical Example: Clustering customer data for segmentation

Deep Learning

This course provides an in-depth exploration of Deep Learning, covering fundamental concepts, state-of-the-art architectures, and practical applications. It introduces students to neural networks, optimization techniques, and specialized models such as CNNs, RNNs, and Transformers.

History and evolution of neural networks

• Why deep learning works today (Data, Compute, and Algorithm improvements)
• Introduction to Perceptron and activation functions (Sigmoid, ReLU, Tanh)
• Gradient Descent and Backpropagation
• Implementing a simple perceptron in Python
• Visualizing activation functions and their derivatives

Deep Neural Networks & Optimization

• Forward and Backward Propagation
• Cost functions (Cross-Entropy, MSE)
• Regularization techniques (L1, L2, Dropout)
• Hyperparameter tuning (Learning Rate, Batch Size, Epochs)
• Gradient Descent Variants: SGD, Momentum, RMSprop, Adam
• Implementing a basic multi-layer perceptron (MLP)
• Using dropout and batch normalization in training

Convolutional Neural Networks (CNNs)

• Why CNNs? Handling high-dimensional image data
• Filters, Kernels, and Convolution operations
• Pooling layers (Max Pooling, Average Pooling)
• Popular CNN Architectures (LeNet, AlexNet, VGG, ResNet)
• Implementing a CNN for image classification
• Training a CNN on MNIST/CIFAR dataset

Recurrent Neural Networks (RNNs) & LSTMs

• Sequence modeling and challenges (Vanishing gradient)
• Recurrent Neural Networks (RNN) basics
• Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU)
• Applications: Time series prediction, NLP
• Implementing a simple RNN
• Training an LSTM model for text generation

Transformers & Attention Mechanisms

• Attention Mechanism & Self-Attention
• Transformer architecture overview
• Pre-trained models: BERT, GPT, and Hugging Face models
• Fine-tuning a Transformer for a custom NLP task

Explainability in AI

• Definition and importance of explainability (trust, fairness, compliance)
• Black-box vs. transparent models
• Real-world case studies: bias in hiring algorithms and criminal justice
• Challenges in interpreting complex models

• Intrinsic (Ante Hoc) vs Post-Hoc methods
• Model-specific vs Model-agnostic
• Global vs Local explanations
• Trade-offs: interpretability vs performance

 Intrinsically Explainable Models (Ante Hoc)

• Decision Trees and their rule-based structure
• Generalized Linear Models (GLMs) and interpretability through coefficients
• Statistical interpretation using t-tests and odds ratios
• Confounding variables and Simpson’s paradox

Global Post-Hoc Explainability

• Permutation Feature Importance (PFI)
• Leave-One-Feature-Out (LOFO) importance
• Partial Dependence Plots (PDP)
• Individual Conditional Expectation (ICE) plots
• Feature interaction via the H-statistic
• Surrogate models to approximate complex black-box models

Local Post-Hoc Explainability

• LIME for tabular, text, and image data
• Shapley Values and their cooperative game theory foundation
• SHAP (SHapley Additive exPlanations) for unified feature attribution
• KernelSHAP for efficient estimation of SHAP values
• SHAP summary and aggregation plots

Model-Specific Explainability Techniques

• GINI importance in Random Forests
• Feature importance in Gradient Boosted Trees (gain, cover, frequency)
• TCAV (Testing with Concept Activation Vectors)
• Saliency Maps for neural networks
• Grad-CAM for CNN visual explanations
• Layer-wise Relevance Propagation (LRP)
• DeepLIFT for back-propagating relevance scores