Events

The role of the algorithm in generalization remains one of the least understood aspects of modern machine learning. Classical theories, such as VC-theory and PAC learning, posits that the sample size needed to fit a model depends only on the class to be learnt but not on the fitting algorithm itself. Yet, in practice, the algorithm plays a crucial role in avoiding overfitting. A well-studied framework to explore this possibility is Stochastic Convex Optimization, where the algorithm's influence on generalization is well established. We will discuss two recent results that try to shed light on how algorithms affect generalization.

The first result examines the sample complexity of Gradient Descent. Arguably this is one of the simplest algorithms for this setup. We will present the first tight sample complexity bounds. These bounds demonstrate how, when applied naively, Gradient Descent performs no better than a worst-case empirical risk minimizer. However, with correct parameter tuning, the algorithm achieves optimal sample complexity rates, but in a computationally inefficient manner.

The second result investigates the interplay between memorization and learning and the potential of information-theoretic generalization bounds. Contrary to the conventional view that successful learning avoids memorization, we will see that even in simple scenarios, memorization can be essential. This finding suggests that large-scale learning might, unintuitively, require complete memorization of the dataset.

Nonlinear activation functions are pivotal to the success of deep neural nets, and choosing the appropriate activation function can significantly affect their performance. Most networks use fixed activation functions (e.g., ReLU, GELU, etc.), and this choice might limit their expressiveness. Furthermore, different layers may benefit from diverse activation functions. Consequently, there has been a growing interest in trainable activation functions. In this paper, we introduce DiTAC, a trainable highly-expressive activation function based on an efficient diffeomorphic transformation (called CPAB). Despite introducing only a negligible number of trainable parameters, DiTAC enhances model expressiveness and performance, often yielding substantial improvements. It also outperforms existing activation functions (regardless whether the latter are fixed or trainable) in tasks such as semantic segmentation, image generation, regression problems, and image classification. The talk is based on [Chelly et. all, ECCV '24].

Paper:

https://arxiv.org/abs/2407.07564

Speakers' short bio:

Irit Chelly is a PhD student in the Computer Science department at Ben-Gurion University, where she also earned her M.Sc., under the supervision of Dr. Oren Freifeld in the Vision, Inference, and Learning group. Her research focuses on probabilistic clustering using non-parametric Bayesian models and unsupervised learning. Her previous projects involved spatial transformations and dimensionality reduction in video analysis, and generative models. Irit won the national-level Aloni PhD scholarship from Israel's Ministry of Technology and Science as well as the BGU Hi-tech scholarship for excellent PhD students, and received annually awards and instructor rank for outstanding teaching skills in essential courses in the Computer Science department. 

Shira Ifergane is an MSc Computer Science student at BGU, working at the Vision, Inference, and Learning group under the supervision of Prof. Oren Freifeld. Shira co-authored an ECCV 2024 paper and has won the national MS scholarship for AI and Data Science research from Israel's Council for Higher Education. Her current research centers on efficient deep models for video analysis.

We provide the final step in the resolution of Bourgain's slicing problem in the affirmative. Thus we establish the following theorem: for any convex body K in R^n of volume one, there exists a hyperplane H, such that the (n-1)-dimensional volume of the intersection of K with H is at least c. Here c > 0 is a universal constant. Our proof combines Milman's theory of M-ellipsoids, stochastic localization with a recent bound by Guan, and stability estimates for the Shannon-Stam inequality by Eldan and Mikulincer.

Joint work with J. Lehec.

Deep neural networks are becoming incredibly sophisticated; they can generate realistic images, engage in complex dialogues, analyze intricate data, and execute tasks that appear almost human-like. But how do such models achieve these abilities?

In this talk, I will present a line of work that aims to explain behaviors of deep neural networks. This includes a new approach for evaluating cross-domain knowledge encoded in generative models, tools for uncovering core mechanisms in large language models, and their behavior under fine-tuning. I will show how to automate and scale the scientific process of interpreting neural networks with the Automated Interpretability Agent, a system that autonomously designs experiments on models’ internal representations to explain their behaviors. I will demonstrate how such understanding enables mitigating biases and enhancing models’ performance. The talk will conclude with a discussion of future directions, including developing universal interpretability tools and extending interpretability methods to automate scientific discovery.

Bio: 

Tamar Rott Shaham is a postdoctoral researcher at MIT CSAIL in Prof. Antonio Torralba’s lab. She earned her PhD from the ECE faculty at the Technion, supervised by Prof. Tomer Michaeli. Tamar has received several awards, including the ICCV 2019 Best Paper Award (Marr Prize), the Google WTM Scholarship, the Adobe Research Fellowship, the Rothchild Postdoctoral Fellowship, the Vatat-Zuckerman Postdoctoral Scholarship, and the Schmidt Postdoctoral Award.

Vizing's Theorem from 1964 states that any n-vertex m-edge graph of maximum degree Δ can be edge colored using at most Δ+1 different colors.

Vizing's original proof is algorithmic and implies that such an edge coloring can be found in O(mn) time.

In this talk, I'll present a randomized algorithm that computes a (Δ+1)-edge coloring in near-linear time -- in fact, only O(mlogΔ) time -- with high probability.

Image generative models are advancing rapidly, producing images of remarkable realism and fidelity. However, existing models often lack precise control over the generated content, limiting their image editing capabilities and the integration of real content into synthesized imagery. In this talk, I will demonstrate how a deep understanding of the inner mechanisms of large-scale pretrained generative models enables the design of powerful techniques for a variety of image manipulation tasks. By analyzing the semantic representations learned by these models, I will present methods that enable effective content editing. Additionally, I will discuss the challenges and trade-offs involved in manipulating real content and propose strategies to address these challenges. Finally, I will highlight recent advancements in incorporating real content, with a particular focus on techniques for injecting information into pretrained models.

Bio: Or Patashnik (https://orpatashnik.github.io/) is a Computer Science PhD candidate at Tel Aviv University, supervised by Daniel Cohen-Or. Her research focuses on computer graphics and its intersection with computer vision, with an emphasis on generative tasks such as image editing, personalization, and image inversion using large-scale pretrained models. Recently, she has been particularly interested in better understanding diffusion models for various applications.

"In the 2004 Olympics, the US national basketball team failed to win the gold medal despite featuring superstars such as LeBron James. This event raises a fundamental question: Why do teams with highly skilled players sometimes underperform compared to teams with less skilled players? In this talk, I will explore situations where such an outcome can be attributed to a surge in free-riding behavior resulting from rational choices by self-interested players. Specifically, I will demonstrate how, under certain incentivizing schemes, improving individual efficiency can inadvertently promote excessive free-riding behavior, ultimately leading to reduced outcomes at both group and individual levels. I will illustrate this phenomenon through two simple games: one on group foraging and the other on workplace cooperation. In the foraging game, an increase in available food can paradoxically lead to a decrease in food consumption, while in the workplace model, replacing workers with more skilled individuals can worsen both individual payoffs and group performance."