Context Vectors Are Reflections of Word Vectors in Half the Dimensions

Simple item page
dc.contributor.authorAssylbekov, Zhenisbek
dc.contributor.authorTakhanov, Rustem
dc.date.accessioned2019-12-11T08:03:54Z
dc.date.available2019-12-11T08:03:54Z
dc.date.issued2019-09
dc.descriptionhttps://arxiv.org/pdf/1902.09859.pdfen_US
dc.description.abstractThis paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.en_US
dc.identifier.citationAssylbekov, Z., & Takhanov, R. (2019). Context Vectors are Reflections of Word Vectors in Half the Dimensions. Journal of Artificial Intelligence Research, 66, 225–242. https://doi.org/10.1613/jair.1.11368en_US
dc.identifier.other10.1613/jair.1.11368
dc.identifier.urihttp://nur.nu.edu.kz/handle/123456789/4369
dc.language.isoenen_US
dc.publisherAI ACCESS FOUNDATIONen_US
dc.rightsAttribution-NonCommercial-ShareAlike 3.0 United States*
dc.rights.urihttp://creativecommons.org/licenses/by-nc-sa/3.0/us/*
dc.subjectContext Vectorsen_US
dc.subjectReflections of Word Vectorsen_US
dc.subjectWord Vectorsen_US
dc.subjectEuclidean normen_US
dc.subjectword2vecen_US
dc.titleContext Vectors Are Reflections of Word Vectors in Half the Dimensionsen_US
dc.typeArticleen_US
workflow.import.sourcescience

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Context Vectors are Reflections of Word Vectors in Half the.pdf
Size:
422.9 KB
Format:
Adobe Portable Document Format
Description:
Article
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
6 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Articles