Figure3:A2Dnon-linearprojection,usingt-SNE[51],ofembeddingsofmethodnamesappearinginmethoddeclarationsintheelasticsearchproject.Similarmethodshavebeengroupedtogether,eventhoughthemodelhasnonotionofthetextualsimilarityofthemethodnames,forexample,theassert-likemethodsontheleftorthenewarraymethodsontheright.betweennearbypoints,orminimizingthedistancebetweeneachpointanditsimageinthe2Dspace.Classicaltechniquesfordimen-sionalityreductionincludeprincipalcomponentsanalysis(PCA)andmultidimensionalscaling.Wewillalsoemployamoremodernmethodcalledt-SNE[51].
Figure3displaysthevectorsassignedtoafewmethodnamesfromatypicalproject(elasticsearch).Eachpointrepresentstheqvectoroftheindicatedtoken.Tointerpretthis,recallthatthemodelusestheqtvectorstopredictwhethertokentwilloccurinparticularcontext.Therefore,tokenstandt withsimilarvectorsqandqtt aretokensthatthemodelexpectstooccurinsimilarcontexts.Theseembeddingsweregeneratedfromthelogbilinearcontextmodel—thatis,withoutusingsubtokens—sothemodelhasnoinformationaboutwhichtokensaretextuallysimilar.Rather,theonlyinformationthatthemodelcanexploitisthecontextsinwhichthetokensareused.Despitethis,wenoticethatmanyofthenameswhicharegroupedtogetherseemtohavesimilarfunctions.Forexample,thereisagroupofassertXXXXmethodsonthelefthandside.Especiallystrikingistheclumpofconstructionmethodsontheright-handsidenewDoubleArray,newIntArray,newLongArray,andsoon.Itisalsotellingthatnearthisclump,thenamesgrowandresizearealsoclosetogether.Analysisrevealsthatthesenamesdoindeedseemtonamemethodsofdifferentclassesthatseemtohavesimilarfunctionality.Ourpreviouswork[2]indicatesthatdevelopersoftenprefersuchentitiestohaveconsistentnames.AdditionallyweexaminethenearestneighborsoftokensintheD-dimensionalspace.Thistypeofanalysisavoidstherisk,inherentinanydimensionalityreductionmethod,thatimportantinformationislostintheprojectionfromDdimensionsto2D.Table2showssomeidenti ersonadifferentproject,clojure,foreachidenti ergivingalistofotheridenti ersthatarenearestinthecontinuousspace.Thenearestneighborsofatokentarethosetokensvsuchthattheinnerproductoftheembeddings,thatis,qt qv,ismaximized.Wechoosethismeasurebecauseitmostcloselymatchesthenotionofsimilarityinthemodel.Again,weareusingthelogbilinearcontextmodelwithoutsubtokeninformation.WeagainseethatthenearestneighborsinthecontinuousspaceseemtohavesimilarsemanticfunctionsuchasthetriplefieldName,methodName,andclassNameorthenamesreturnType,typ,andtype.
Table3takesthisanalysisabitfurtherforthesubtokenmodel.Thistableshowsthe“nearestnearestneighbors”:thosepairsoftokensorsubtokensthatareclosestintheembeddingspaceoutofallpossiblepairsoftokens.Ontheleftcolumn,weseepairsofcloseneighborsfromthefeature-basedbilinearcontextmodelwithoutsubtokens.Thesecontainmanysimilarpairs,suchaswidthandheight.Itisstrikinghowmanyofthesepairscontainsimi-larsubtokenseventhoughthismodeldoesnotcontainsubtokens.Movingtothesubtokenmodel,therightcolumnofTable3showspairsofsubtokensthatareclosestintheembeddingspace.Themodellearnsthatpairslikenumerals,MinandMax,andHeight
andWidthshouldbeplacedneartoeachotherinthecontinuousspace.Thisisfurtherevidencethatthemodelislearningsemanticsimilaritiesgivenonlystatisticalrelationshipsbetweentokens.Wecanalsoattempttobeabitmorespeci cinouranalysis.InthisweareinspiredbyMikolovetal.[35],whonoticedthataddingtogethertwooftheirembeddingsofnaturallanguagewordsoftenyieldedacompositionalsemantics—e.g.embedding(“Paris”)-em-bedding(“France”)+embedding(“Vietnam”)yieldedavectorwhosenearestneighborwastheembeddingof“Hanoi”.Toattemptsome-thingsimilarforsourcecode,weconsidersemanticrelationshipsthatpairsofidenti ershavewitheachother.
ForFigures4and5,weprojecttheD-dimensionalembeddingsto2DusingPCAratherthant-SNE.Althoughatechnicalpoint,thisisimportant.Unliket-SNE,PCAisalinearmethod,thatis,themappingbetweentheD-dimensionalpointsandthe2Dpointsislinear.Therefore,ifgroupsofpointsareseparatedbyaplaneinthe2Dspace,thenweknowthattheyareseparatedbyaplaneinthehigher-dimensionalspaceaswell.Figure4showstheembeddingsofallpairsofsetterandgettermethodsfortheprojectnetty.Thesubtokenmodeldidnotgeneratethesemodels,sothemodelcannotclusterthesetokensbasedontextualsimilarity.Nevertheless,we ndthatgetterandsettertokensarereasonablywellseparatedinthecontinuousspace,becausetheyareusedinsimilarcontexts.InFigure5,wematchpairsofvariablenamesinthelibgdxprojectinwhichonename(the“pluralname”)equalsanothername(the“singularname”)plusthecharacters.TheJavaconventionthatCollectionobjectsareoftennamedbypluralvariablenamesmotivatesthischoice.Althoughthismappingismorenoisythanthelast,westillseethatpluralnamestendtoappearontheleftsideofthe gure,andsingularnamesontheright.
Fromthisexplorationweconcludethatthecontinuouslocationsofeachnameseemtobecapturingsemanticregularities.Readerswhowishtoexplorefurthercanviewtheembeddingsathttp://groups.inf.ed.ac.uk/cup/naturalize/.
Eventhoughthecontinuousembeddingsarelearnedfromcon-textalone,thesevisualizationssuggestthattheseembeddingsalsocontain,tosomeextent,semanticinformationaboutwhichiden-ti ersaresimilar.Thissuggeststhatlocalandglobalcontextdoprovideinformationthatcanberepresentedandexploited,thatis,semanticallysimilarnamesareusedinsimilarcontexts.Thisisevidencepointingtowardsanaf rmativeanswertoRQ1.Itisespe-ciallystrikingthatwehaveconsistentlyfoundthatnearbytokensinthecontinuousspacetendtosharesubtokens,evenwhenthemodeldoesnotincludesubtokeninformation.TherightcolumnofTable3reinforcesthispointsinceitshowsthat,whenwedousethesubtokenmodel,nearbypairsofsubtokensinthecontinuousspaceseemtobemeaningfullyrelated.Thisprovidessomeevidenceforanaf rmativeanswertoRQ2.
Finally,itcanbeobjectedthatthistypeofanalysisisnecessarilysubjective.Whenbackedandvalidatedbyquantitativeanalysis,