Abstract

We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.

Keywords

Computer scienceParaphraseSentenceArtificial intelligenceNatural language processingVocabularyEncoderBenchmark (surveying)ENCODERanking (information retrieval)Speech recognitionLinguistics

Affiliated Institutions

Related Publications

DeepWalk

Bryan Perozzi , Rami Al-Rfou , Steven Skiena

We present DeepWalk, a novel approach for learning latent representations of\nvertices in a network. These latent representations encode social relations in\na continuous vector...

2014 8168 citations

Publication Info

Year
2015
Type
article
Volume
28
Pages
3294-3302
Citations
723
Access
Closed

External Links

View on DOI.org

Social Impact

Altmetric
PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

723
OpenAlex

Cite This

Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov et al. (2015). Skip-Thought Vectors. arXiv (Cornell University) , 28 , 3294-3302. https://doi.org/10.48550/arxiv.1506.06726

Identifiers

DOI
10.48550/arxiv.1506.06726