BENIGN OVERFITTING WITH RETRIEVAL AUGMENTED MODELS

Simple item page
dc.contributor.authorAssylbekov, Zhenisbek
dc.contributor.authorTezekbayev, Maxat
dc.contributor.authorNikoulina, Vassilina
dc.contributor.authorGallé, Matthias
dc.date.accessioned2022-10-11T08:54:20Z
dc.date.available2022-10-11T08:54:20Z
dc.date.issued2022
dc.description.abstractDespite the fact that modern deep neural networks have the ability to memorize (almost) the entire training set they generalize well to unseen data, contradicting traditional learning theory. This phenomenon --- dubbed benign overfitting --- has been theoretically studied so far in simplified settings only. At the same time, ML practitioners (especially in NLP) figured out how to exploit this feature for more efficient training: retrieval-augmented models (e.g., kNN-LM, RETRO) explicitly store (part of) the training sample in the storage and thus try to (partially) remove a load of memorization from the neural network. In this paper we link these apparently separate threads of research, and propose several possible research directions regarding benign overfitting in retrieval-augmented models.en_US
dc.identifier.urihttp://nur.nu.edu.kz/handle/123456789/6735
dc.language.isoen_USen_US
dc.publisherNazarbayev University School of Sciences and Humanities
dc.rightsCC0 1.0 Universal*
dc.rights.urihttp://creativecommons.org/publicdomain/zero/1.0/*
dc.subjectMachine Learning, Retrieval Augmentation, Benign Overfitting, Long Tail Theory, Simplicity Bias, Natural Language Processing, Memorization and Generalization, Learning Theoryen_US
dc.subjectType of access: Open Access
dc.titleBENIGN OVERFITTING WITH RETRIEVAL AUGMENTED MODELSen_US
dc.typeWorking Paperen_US
workflow.import.sourcescience

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Benign_Overfitting_with_Retrieval_Augmented_Models.pdf
Size:
489.34 KB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
6.28 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Working papers