Figureofgetter/settingsetters4:Aand2Dlinearprojection,usingPCA,oftheembeddings
beddingsseempairsgetterstoseparateareforconnectednettymethodsettersfromwithdeclarations.theadottedgetters.
line.MatchedTheem-Tableforvariable2:Examplesnamesofinnearestclojure.neighborsOrderedbyinhigherthecontinuousinnerproductspaceqt 1qt2
wheret1isinthe rstcolumnandt2inthesecond.Identi er
NearestNeighbors(orderedbydistance)
methods
,,,,returnTypekeyvalssbparamsitems,typ,type,methodName,t
paramType,seq,form,rest,valOrNode
,ctor,methodName,args,arg
however,thisanalysisprovidesvisualinsight,gainedfromlookingattheembeddingvectors.Thus,wecomplementthisqualitativeanalysiswithamorequantitativeone,inthenextsection.
5.EVALUATION
Inthissection,wequantitativelyevaluatetheperformanceoftheneuralmodelonthedataset(Table1)answeringalltheRQs.VariableNamingRenamingvariablesandmethodinvocationshasbeenpreviouslyshown[2]toachievegoodperformanceusingn-gramLMs.Figure6showstheperformanceofthebaselinen-grammodelalongwiththeperformanceoftheotherneuralmodelsforvariablenames.Forlowfrequencyofsuggestions(highcon dencedecisions),theneuralmodelsoverperformthen-gram-basedsug-gestions.Thisisexpectedsincesuchmodelsperformbetterthanplainn-grammodelsinNLP[39].Additionally,thefeaturesgiveasubstantialperformanceincreaseoverthemodelsthatlackfeatures.Thesubtokenmodelperformsworsecomparedtothetoken-levelmodelforsuggestionfrequencieshigherthan6%.Thisistobeexpected,sincethesubtokenmodelhastomakeasequenceofincreasinglyuncertaindecisions,predictingeachsubtokensequen-tially,increasingthepossibilityofmakingamistakeatsomepoint.Forsuggestionfrequencieslowerthan6%theperformanceofthesubtokenmodelisslightlybettercomparedtothetoken-levelmodel,thankstoitsabilitytogeneratenovelidenti ers.Thus,wepositivelyanswerRQ1andRQ2.
WecomputedTable4overonlythreeclassesbecauseofthecostofretrainingthemodelonefeatureatatime.LookingatTable4forvariablenamesonemayseehoweachfeatureaffectstheperformanceofthemodelsoverthebaselineneuralmodelwithnofeaturesatrankk=52.First,weobservethatthefeatures
2We
chose vebecausesubitizing,theabilitytocountataglance,
Figureof5:A2Dlinearprojection,usingPCA,oftheembeddings
withsingularandaandpluralnamesinlibgdx.PairsareconnectedfertopluraldottedCollectionsnames.line.WeTheofexpectembeddingsmostlyseparatesingularobjectsmostwhoseofnamesthepluralappearvariablesinsingular.tore-Table3:Closelyrelated(sub-)tokensforlibgdxvariables.Thetop10pairsthathavethehighestqpairs(t 1qtareshown.Forthesubtokenmodelsomenumerale.g.2
9–8)areomitted.
FeatureModel
Subtoken
padBottom–dataOut–padLeftHeight–localAnchorA–dataIn
swig–WidthbodyA–localAnchorBMin–classframebuffers–bodyB
shape–Max
worldWidth–buffersLeft–collisionpadRight–worldHeightcamera–Rightjarg7–padLeftTOUCH–camend–KEYspriteBatch–jarg6_
–batchloc–start–location
helpmostlyathighsuggestionfrequencies.Thisisduetothefactthatforhigh-con dence(lowsuggestionfrequency)decisionthemodelsarealreadygoodatpredictingthosenames.Additionally,combiningallthefeaturesyieldsaperformanceincrease,suggestingthatforvariablenames,onlythecombinationofthefeaturesgivessuf cientlybetterinformationaboutvariablenaming.
MethodDeclarationNamingAccuracyWenowattempttousetheneuralmodelforsuggestingmethodnames,usingonlyfeaturesavailableduringthedeclarationofamethod.Surprisingly,theneuralmodelisexceptionallygoodatpredictingmethoddeclarationnames.Figure7ashowstheperformanceofthemodelsonallmethoddeclarationsexcludinganymethoddeclarationsthataremethodoverrides.WeexcludeoverridessoastoavoidgivingthemodelscreditforpredictingeasynamesliketoString.Whenweincludeoverrides,theperformanceofallmodelsimproves.Toexcludemethodoverrides,weremovemethodsthatcontainthe@Overrideannotationaswellasthosemethodsthatwecanstaticallydetermineasmethodoverrides.
ThegraphsinFigures7ashowthattheneuralmodelsaresubstan-tiallybetteratsuggestingmethodnames,comparedtothen-gramlanguagemodel.Addingfeaturesincreasestheperformanceofthemodels,indicatingthatthemodelisabletousenon-localcontexttomakebetterpredictions.Naturally,theperformancedegradeshandlesthesize5ofobjectshumanandshortbecausetermmemory.
shorttermmemoryisusually7±2is