Table1:Evaluationprojects(Java).Orderedbypopularity.
Name
GitSHADescription
elasticsearch
d3e10f9RESTSearchEngineAndroid-Universal-Image-Loader19ce389AndroidLibrary
spring-framework2bf6b41ApplicationFrameworklibgdx789aa0bGameDevFrameworkstormbc54e8eDistributedComputationzxing71d8395Barcodeimageprocessingnetty
3b1f15eNetworkAppFrameworkplatform_frameworks_basef19176fAndroidBaseFrameworkbigbluebutton02bc78cWebConferencingjunitc730003TestingFramework
rxjavacf5ae70ReactiveJVMextensionsretro tafd00fdRESTclient
clojuref437b85ProgrammingLanguagedropwizard741a161RESTfulwebserverokhttp0a19746HTTP+SPDYclientpresto6005478DistributedSQLenginemetrics4984fb6MetricsFramework
spring-bootb542aa3AppFrameworkWrapperbukkitf210234MincraftModAPI
nokgiri
a93cde6
HTML/XML/CSSparser
Inreferencetothe rsttworesearchquestionswedescribethefeaturesthatweuseandhowwecapturesubstructureinthenextsection.Wethende nitivelyanswertheseresearchquestionsbycomparingourapproachwithprevioustechniquesforsuggestingvariablenamesonabroadsoftwarecorpus.Thereislittletonoresearchthattacklesthesecondtworesearchquestionstocompareagainst.Nonetheless,weuseann-grammodelasapointofcompari-sonfornamingmethodsandclassestodemonstratetheperformanceofourapproachasthatmodelhasperformedthebestforvariablenaminginthepastandwehypothesizeisreasonableforthesenewnamingtasks.Therestofthissectiondescribesourexperimentalsetupandmethodology.
DataWepickedthetopactiveJavaGitHubprojectsonJanuary22nd2015.Weobtainedthemostpopularprojectsbytakingthesumofthez-scoresofthenumberofwatchersandforksofeachproject,usingtheGitHubArchive.Startingfromthetopproject,weselectedthetop20projectsexcludingprojectsthatwereinadomainthatwaspreviouslyselected.Wealsoincludedonlyprojectswithmorethan50collaboratorsandmorethan500commits.TheprojectsalongwithshortdescriptionsareshowninTable1.Weusedthisproceduretoselectamature,active,anddiversecorpuswithlargedevelopmentteams.Finally,wesplitthe lesuniformlyintoatraining(70%)andatest(30%)set.
MethodologyWetrainallmodelsonthetrainsetsformedoverthe lesofeachproject.Toevaluatethemodels,foreachofthetest lesandforeachvariable(allidenti ersthatresolvetothesamesymbol),methoddeclarationorclassdeclaration,wecomputethefeaturesandcontextofthelocationoftheidenti erandaskthemodeltopredicttheactualtargettokenthedeveloperused(whichisunknowntothemodel),asinAllamanisetal.[2].Intheusecasesweconsider(seeSection1),modelsthataredeployedwitha“con dence lter”,thatis,themodelwillonlypresentasuggestiontotheuserwhentheprobabilityofthetoprankednameisabovesomethreshold.Thisistoavoidannoyingtheuserwithlow-qualitysuggestions.Tore ectthisinourevaluation,wemeasurethedegreetowhichthequalityoftheresultschangesasafunctionofthethreshold.Ratherthanreportingthethreshold,whichisnotdirectlyinterpretable,weinsteadreportthesuggestionfrequency,whichisthepercentageofnamesinthetestsetforwhichthemodeldecidestomakeapredictionforagiventhreshold.
Tomeasurethequalityofasuggestion,wecomputetheF1scoreandtheaccuracyfortheretrievaloverthesubtokensofeachcorrecttoken.Thus,allmethodsaregivenpartialcreditifthepredictednameisnotanexactmatchbutsharessubtokenswiththecorrectname.F1istheharmonicmeanofprecisionandrecallandisastandardmeasure[32]becauseitconservative:asaharmonicmean,
itsvalueisin uencedmostbythelowestofprecisionandrecall.Wealsocomputetheaccuracyofeachprediction:apredictioniscorrectwhenthemodelpredictsexactly(exactmatch)theactualtoken.WhencomputingtheF1scoreforsuggestionrankk>1,wepicktheprecision,recall,andF1oftherankl≤kthatresultsinthehighestF1score.
Becausethisevaluationfocusesonpopularprojects,theresultsmaynotre ectperformanceonalowqualityprojectinwhichmanynamesarepoor.Forsuchprojects,werecommendtrainingonadifferentprojectthathashighqualitynames,butleaveevaluatingthisapproachtofuturework.Alternatively,onecouldarguethat,becausewemeasurewhetherthemodelcanreproduceexistingnames,theevaluationistooharsh:ifapredictednamedoesnotmatchtheexistingname,itcouldbeequallygood,orevenanimprovement.Nonetheless,matchingexistingnamesinhighqualityprojects,aswedo,stillprovidesevidenceofoverallsuggestionquality,and,comparedtoauserstudy,anautomaticevaluationhasthegreatadvantagethatitallowsef cientcomparisonofalargernumberofdifferentmethods.
Finally,duringtraining,wesubstituterareidenti ers,subtokensandfeatures(i.e.thoseseenlessthantwotimesinthetrainingdata)withaspecialUNKtoken.Duringtesting,whenanyofthemodelssuggeststheUNKtoken,wedonotmakeanysuggestions;thatis,theUNKtokenindicatesthatthemodelexpectsaneologismthatitcannotpredict.Forthesubtokenmodel,duringtesting,wemayproducesuggestionsthatcontainUNKsubtokens.IntheunlikelycasethatacontexttokentwithaspecialSi+k=ti(i.e.isthesametoken),wereplaceti+kELFtoken.Thismakessurethatthecontextofthemodelincludesnoinformationaboutthetargettoken.
TrainingParametersWeusedlearningrate0.07,D=50,mini-batchsize100,dropout20%[48],generated10distractorsforeachsampleforeachepochandtrainedforamaximumof25epochspickingtheparametersthatachievedthemaximumlikelihoodinaheldoutvalidationset(the10%ofthetrainingdata).ThecontextsizewassettoK=6andsubtokencontextsizewassettoM=3.Beforethetrainingstarted,parameterswere 4initializedaround0withuniformadditivenoise(scaledby10).ThebiasparametersbwereinitializedsuchthatP(t|c)matchestheempirical(unigram)distributionofthetokens(orsubtokensforthesubtokenmodel).AllthehyperparametersexceptforDweretunedusingBayesianoptimizationonbigbluebuttonformethoddeclarations.Thepa-rameterDisspecialinthatasweincreaseit,theperformanceofeachmodelincreasesmonotonically(assumingagoodvalidationset),withdiminishingreturns.Also,anincreaseinDincreasesthecomputationalcomplexityoftrainingandtestingeachmodel.WepickedD=50thatresultedinagoodtrade-offofthecomputationalcomplexityvs.performance.
4.IDENTIFIERREPRESENTATION
Firstweevaluateourmodelqualitatively,byvisualizingitsoutput.Allofthemodelsthatwehavedescribedassigntokens,features,andsubtokenstoembeddings,whicharelocationsinaD-dimensionalcontinuousspace.Theselocationshavebeenselectedbythetrain-ingproceduretoexplainstatisticalpropertiesoftokens,butitdoesnotnecessarilyfollowthattheembeddingscaptureanythingaboutthesemanticsofnames.Toexplorethisquestion,weexaminequalitativelywhethernamesthatappearsemanticallysimilartousareassignedtosimilarembeddings,byvisualizingthecontinuousembeddingsassignedtonamesfromafewprojects.Thisraisestheimmediatedif cultyofhowtovisualizevectorsinaD=50dimensionalspace.Fortunately,thereisarichliteratureinstatisticsandmachinelearningaboutdimensionalityreductionmethodsthatmaphighdimensionalvectorstotwo-dimensionalvectorswhilepre-servingimportantpropertiesoftheoriginalspace.Therearevariousideasbehindsuchtechniques,suchaspreservingdistancesorangles