Abstract
We present a new type of neural probabilistic language model that learns a mapping from both words and explicit word features into a continuous space that is then used for word prediction. Additionally, we investigate several ways of deriving continuous word representations for unknown words from those of known words. The resulting model significantly reduces perplexity on sparse-data tasks when compared to standard backoff models, standard neural language models, and factored language models.
Keywords
PerplexityLanguage modelComputer scienceWord (group theory)Artificial intelligenceNatural language processingProbabilistic logicArtificial neural networkLinguistics
Affiliated Institutions
Related Publications
Enriching Word Vectors with Subword Information
Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore ...
Recurrent Continuous Translation Models
We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words,...
Improving Word Representations via Global Context and Multiple Word Prototypes
Unsupervised word representations are very useful in NLP tasks both as inputs to learning algorithms and as extra word features in NLP systems. However, most of these models are...
Glove: Global Vectors for Word Representation
Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the o...
Better Word Representations with Recursive Neural Networks for Morphology
Vector-space word representations have been very successful in recent years at improving performance across a variety of NLP tasks. However, common to most existing work, words ...
Publication Info
- Year
- 2006
- Type
- article
- Pages
- 1-4
- Citations
- 95
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
95
OpenAlex
Cite This
Andrei T. Alexandrescu,
Katrin Kirchhoff
(2006).
Factored neural language models.
, 1-4.
https://doi.org/10.3115/1614049.1614050
Identifiers
- DOI
- 10.3115/1614049.1614050