Jan Hendrik Metzen

I am a Research Scientist at Prior Labs, where I develop the next generation of tabular foundation models. My work focuses on scalable pretraining, novel architectures, and scaling laws. I contribute to TabPFN-3, which scales tabular foundation models to datasets with up to one million training rows while substantially accelerating training and inference.

Previously, I was a Senior AI Researcher in Aleph Alpha Research’s Foundation Models team, working on efficient LLM pretraining, tokenizer-free architectures, and large-scale optimization. Before that, I was a Senior Expert at the Bosch Center for Artificial Intelligence, where my research centered on robust and reliable computer vision, neural architecture search, and synthetic data.

My broader interests span efficient and reliable machine learning, AutoML, and open-source software. I am an ELLIS member, an ELIZA Industrial Fellow, and an Area Chair for NeurIPS 2026. I have also contributed to scikit-learn, including its probability-calibration, kernel-ridge, and Gaussian-process modules.

Research interests

Tabular foundation models · Scalable pretraining · Efficient architectures · Scaling laws · Reliable machine learning

Publications Curriculum vitae Google Scholar GitHub

news

May 13, 2026	We released the TabPFN-3 technical report, scaling tabular foundation models to datasets with up to one million training rows.
Mar 16, 2026	As part of my previous role at Aleph Alpha Research, I contributed to A Family of LLMs Liberated from Static Vocabularies, presenting tokenizer-free architectures for efficient pretraining, adaptation, and inference.
Feb 1, 2026	I joined Prior Labs as a Research Scientist, working on the next generation of tabular foundation models.
Aug 12, 2024	I launched SVGStud.io, an AI-powered tool for searching, generating, and customizing scalable vector graphics.

selected publications

2026

TabPFN-3: Technical Report

Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Manium, Shi Bin Hoo, Magnus Bühler, Anurag Garg, Dominik Safaric, Jake Robertson, Benjamin Jäger, Simone Alessi, Adrian Hayler, Vladyslav Moroshan, Lennart Purucker, Philipp Singer, Alan Arazi, Julien Siems, Jan Hendrik Metzen, Georg Grab, Nick Erickson, Siyuan Guo, Eliott Kalfon, Simon Bing, David Salinas, Clara Cornu, Lilly Charlotte Wehrhahn, Diana Kriuchkova, Kursat Kaya, Lydia Sidhoum, Marie Salmon, Jerry Chen, Madelon Hulsebos, Yann LeCun, Samuel Müller, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter

2026

arXiv PDF

2024

AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models

Jan Hendrik Metzen, Piyapat Saranrittichai, and Chaithanya Kumar Mummadi

Transactions on Machine Learning Research, 2024

arXiv PDF

Classifiers built upon vision-language models such as CLIP have shown remarkable zero-shot performance across a broad range of image classification tasks. Prior work has studied different ways of automatically creating descriptor sets for every class based on prompt templates, ranging from manually engineered templates over templates obtained from a large language model to templates built from random words and characters. Up until now, deriving zero-shot classifiers from the respective encoded class descriptors has remained nearly unchanged, i.e., classify to the class that maximizes cosine similarity between its averaged encoded class descriptors and the image encoding. However, weighing all class descriptors equally can be suboptimal when certain descriptors match visual clues on a given image better than others. In this work, we propose AutoCLIP, a method for auto-tuning zero-shot classifiers. AutoCLIP tunes per-image weights to each prompt template at inference time, based on statistics of class descriptor-image similarities. AutoCLIP is fully unsupervised, has very low computational overhead, and can be easily implemented in few lines of code. We show that AutoCLIP outperforms baselines across a broad range of vision-language models, datasets, and prompt templates consistently and by up to 3 percent point accuracy.
Label-free Neural Semantic Image Synthesis

Jiayi Wang, Kevin Alexander Laube, Yumeng Li, Jan Hendrik Metzen, Shin-I Cheng, Julio Borges, and Anna Khoreva

In European Conference on Computer Vision (ECCV), 2024

arXiv PDF

Recent work has shown great progress in integrating spatial conditioning to control large, pre-trained text-to-image diffusion models. Despite these advances, existing methods describe the spatial image content using hand-crafted conditioning inputs, which are either semantically ambiguous (e.g., edges) or require expensive manual annotations (e.g., semantic segmentation). To address these limitations, we propose a new label-free way of conditioning diffusion models to enable fine-grained spatial control. We introduce the concept of \emphneural semantic image synthesis, which uses neural layouts extracted from pre-trained foundation models as conditioning. Neural layouts are advantageous as they provide rich descriptions of the desired image, containing both semantics and detailed geometry of the scene. We experimentally show that images synthesized via neural semantic image synthesis achieve similar or superior pixel-level alignment of semantic classes compared to those created using expensive semantic label maps. At the same time, they capture better semantics, instance separation, and object orientation than other label-free conditioning options, such as edges or depth. Moreover, we show that images generated by neural layout conditioning can effectively augment real data for training various perception tasks.
Feature Distillation Improves Zero-Shot Transfer from Synthetic Images

Niclas Popp, Jan Hendrik Metzen, and Matthias Hein

Transactions on Machine Learning Research, 2024

arXiv PDF

2023

Identification of Systematic Errors of Image Classifiers on Rare Subgroups

Jan Hendrik Metzen, Robin Hutmacher, N Grace Hua, Valentyn Boreiko, and Dan Zhang

In International Conference on Computer Vision, 2023

arXiv PDF

Despite excellent average-case performance of many image classifiers, their performance can substantially deteriorate on semantically coherent subgroups of the data that were under-represented in the training data. These systematic errors can impact both fairness for demographic minority groups as well as robustness and safety under domain shift. A major challenge is to identify such subgroups with subpar performance when the subgroups are not annotated and their occurrence is very rare. We leverage recent advances in text-to-image models and search in the space of textual descriptions of subgroups ("prompts") for subgroups where the target model has low performance on the prompt-conditioned synthesized data. To tackle the exponentially growing number of subgroups, we employ combinatorial testing. We denote this procedure as PromptAttack as it can be interpreted as an adversarial attack in a prompt space. We study subgroup coverage and identifiability with PromptAttack in a controlled setting and find that it identifies systematic errors with high accuracy. Thereupon, we apply PromptAttack to ImageNet classifiers and identify novel systematic errors on rare subgroups.

2022

Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness

Giulio Lovisotto, Nicole Finnie, Mauricio Munoz, Chaithanya Kumar Mummadi, and Jan Hendrik Metzen

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

arXiv PDF

Neural architectures based on attention such as vision transformers are revolutionizing image recognition. Their main benefit is that attention allows reasoning about all parts of a scene jointly. In this paper, we show how the global reasoning of (scaled) dot-product attention can be the source of a major vulnerability when confronted with adversarial patch attacks. We provide a theoretical understanding of this vulnerability and relate it to an adversary’s ability to misdirect the attention of all queries to a single key token under the control of the adversarial patch. We propose novel adversarial objectives for crafting adversarial patches which target this vulnerability explicitly. We show the effectiveness of the proposed patch attacks on popular image classification (ViTs and DeiTs) and object detection models (DETR). We find that adversarial patches occupying 0.5% of the input can lead to robust accuracies as low as 0% for ViT on ImageNet, and reduce the mAP of DETR on MS COCO to less than 3%.