neuralprobabilisticlanguagemodellingandfollowitwithtwonovelmodelsthat,speci callydesignedformethodnaming,re netheunderlyingneuralmodel:ourlogbilinearcontextmodel,whichaddscontextandfeatures,andsubtokencontextmodel,whichaddssubtokensandcanbeusedtogenerateneologisms.
Languagemodels(LM)areprobabilitydistributionsoverstringsofalanguage.Thesemodelsassumethatwearetryingtopredictatokentgivenasequenceofothertokensc=(cwecallthecontext.LMsareverygeneral;forexample,0,c1,...ciftheN)thatgoalistosequentiallypredicteverytokenina le,asan-grammodeldoes,thenwecantaket=ymandc=(ym n+1ym n+2...ym 1).Alternately,forthemethodnamingproblem,wecantakettobetheidenti ertokeninthedeclarationthatnamesthefunction,andctobeasequencethatcontainsallidenti ersinthefunctionbody.Obvi-ously,wecannotstoreaprobabilityvalueforeverypossiblecontext,sowemustmakesimplifyingassumptionstomakethemodelingtractable.DifferentLMsmakedifferentsimplifyingassumptions.
2.1Background
Tobuildintuition,webeginbyreviewingthen-gramLM,which
isastandardtechniqueinNLPandspeechprocessing,andwhichhasbecomeincreasinglypopularinsoftwareengineering[25,41,2].Then-grammodelassumesthatalloftheinformationrequiredtopredictthenexttokeniscontainedwithinthepreviousn 1tokensi.e.P(y1...yM)=∏Mm=1P(ym|ym 1...yVnmnumbers, n+1).Tospecifythismodelweneed(inprinciple)atableofwhereVisthenumberofpossiblelexemes,thatspeci estheconditionalprobabilitiesforeachpossiblen-gram.Thesearetheparametersofthemodelthatwelearnfromdata.
Thereisalargeliteratureonmethodsfortrainingthesemod-els[16],whichbasicallyrevolvearoundcountingtheproportionoftimesthattokenymfollowsym 1...ym n+1.However,evenwhenn=4orn=5,wecannotexpecttoestimatethecountsofalln-gramsreliably,asthenumberofpossiblen-gramsisexponentialinn.Therefore,smoothingmethodsareemployed,whichgenerallymodifythecountofararen-gramythecountofashortersuf xy1...ywhosentomakeitmoresimilarto2...yn,frequencywecanestimatemorereliably.Thisprocedureinvolvestheimplicitassump-tionthattwocontextsaremostsimilariftheysharealongsuf x.Butthisassumptiondoesnotalwayshold.Manysimilarcontexts,suchasx+yversusx+z,mightbetreatedverydifferentlybyan-grammodel,becausethe naltokenisdifferent.
LogbilinearmodelsNeuralLMs[10]addressthechallengethatthesimplen-grammodelhasbymakingsimilarpredictionsforsimilarcontexts.Theypredictthenexttokenyprevioustokensasinput.musinganeuralnetworkthattakestheThisallowsthenetworkto exiblylearnwhichtokens,likeint,providemuchinformationabouttheimmediatelyfollowingtoken,andwhichtokens,likethesemicolon’;’,provideverylittle.Unlikeann-grammodel,aneuralLMmakesiteasytoaddgenerallong-distancefeaturesofthecontextintotheprediction—wesimplyaddthemasadditionalinputstotheneuralnet.Inourwork,wefocusonasimpletypeofneuralLMthathasbeeneffectiveinpractice,namely,thelog-bilinearLM[37](LBL).Westartwithageneraltreatmentofloglinearmodelsconsideringmodelsoftheform
P(t|c)=
exp(sθ(t,c))
∑exp(s,c))
.
(1)
t θ(tIntuitively,sθisafunctionthatindicateshowmuchthemodellikes
toseebothtandctogether,theexpfunctionmapsthistobealwayspositive,andthedenominatorensuresthattheresultisaprobabilitydistribution.Thischoiceisverygeneral.Forexample,ifsfeaturesinc,thenthediscriminativemodelθisalinearfunctionoftheissimplyalogisticregression.
Logbilinearmodelslearnamapfromeverypossibletargetttoa
vectorqt∈RD,andfromeachcontextctoavector r
interprettheseaslocationsofeachcontextandeachtargetc∈RD.WelexemeinaDdimensionalspace;theselocationsarecalledembeddings.Themodelpredictsthatthetokentismorelikelytoappearincontextc
iftheembeddingqtofthetokenissimilartothat r
Toencodethisinthemodel,wechoose
cofthecontext.sθ(t,c)= r cqt+bt,
(2)
wherebtisascalarbiaswhichrepresentshowcommonlytoccursregardlessofthecontext.Tounderstandthisequationintuitively,
notethat,ifthevectors r
thecosineofthecandqanglethavenorm1,thentheirdotproductissimplybetweenthem.Sosvectorhasalargenorm,ifbθ,andhencep(t|c),islargerifeither r
iftislarge,orifcandqthaveasmallanglebetweenthem,thatis,theyaremoresimilaraccordingtothecommonlyusedcosinesimilaritymetric.Tocompletethisdescription,wede nethemapst→qc→r
.Forthetargetst,themostcommonchoiceistosimplytand includecthevectorqtforeverytasaparameterofthemodel.Thatis,thetrainingprocedurehasthefreedomtolearnanarbitrarymapbetweentandqt.Forthecontextsc,thischoiceisnotpossible,astherearetoomanypossiblecontexts.Instead,acommonchoice
[31,39]istorepresenttheembedding r
embeddingsofthetokenswithinit,thatcofacontextasthesumofis,
|C|
r
c=Ctrct
,
(3)
t∑=1
wherercDt∈Risavectorforeachlexemethatisincludedinthemodelparameters.Thevariabletindexeseverytokeninthecontextc,soifthesamelexemeoccursmultipletimesinc,thenitappearsmultipletimesinthesum.ThematrixCdependingontisadiagonalmatrixthatservesasascalingfactorthepositionofalexemewithinthecontext.Thisallows,forexample,alexeme’sin uenceonc’spositiontodependonhowcloseitistothetarget.TheDnon-zerovaluesinClexemetforeachtarealsoincludedinthemodelparameters.Eachvhastwoembeddings:anembeddingqvforwhenitisusedasatargetandanembeddingrvforwhenitappearsinthecontext.
Tosummarize,logbilinearmodelsmaketheassumptionthateverytokenandeverycontextcanbemappedinaD-dimensionalspace.Therearetwokindsofembeddingvectors:thosedirectlylearned(i.e.theparametersofthemodel)andthosecomputedfromtheparametersofthemodel.Toindicatethisdistinction,weplacea
haton r
ctoindicatethatitiscomputedfromthemodelparameters,whereaswewriteqtwithoutahattoindicatethatitisaparametervectorthatislearneddirectlybythetrainingprocedure.Thesemod-elscanalsobeviewedasathree-layerneuralnetwork,inwhichtheinputlayerencodesallofthelexemesincusinga1-of-Vencoding,thehiddenlayeroutputsthevectorsrctforeachtokeninthecontext,andtheoutputlayercomputesthescorefunctionssasoftmaxnonlinearity.Fordetailsontheθ(t,c)andpassesthemtoneuralnetworkrepresentation,seeBengioetal.[10].
Tolearntheseparameters,ithasrecentlybeenshown[39,38]thatanalternativetothemaximumlikelihoodmethodcallednoisecontrastiveestimation(NCE)[21]iseffective.NCEmeasureshowwellthemodelp(t|c)candistinguishtherealdatainthetrainingsetfrom“fantasydata”thatisgeneratedfromasimplenoisedistribution.Atahighlevel,thiscanbeviewedasablackboxalternativetomaximumlikelihoodthatmeasureshowwellthemodel tsthetrainingdata.Weoptimizethemodelparametersusingstochasticgradientdescent.WeemployNCEforallmodelsinthispaper.
2.2LogbilinearContextModelsofCode
Nowwepresentanewneuralnetwork,anovelLBLLMforcode,whichwecallalogbilinearcontextmodel.Thekeyidea