Skip to content
/ models Public

Machine learning models for MLonCode trained using the source{d} stack

Notifications You must be signed in to change notification settings

src-d/models

Repository files navigation

source{d} MLonCode models

bot-detection

Model that identifies bots from humans among developer identities.

Example:

from sklearn.preprocessing import LabelEncoder
from sourced.ml.models import BotDetection
from xgboost import XGBClassifier

bot_detection = BotDetection.load(bot-detection)
xgb_cls = XGBClassifier()
xgb_cls._Booster = bot_detection_model.booster
xgb_cls._le = LabelEncoder().fit([False, True])
print('model configuration: ', xgb_cls)
print('BPE model vocabulary size: ', len(bot_detection.bpe_model.vocab()))

1 model:

bow

Weighted bag-of-words, that is, every bag is a feature extracted from source code and associated with a weight obtained by applying TFIDF.

Example:

from sourced.ml.models import BOW
bow = BOW().load(bow)
print("Number of documents:", len(bow))
print("Number of tokens:", len(bow.tokens))

4 models:

docfreq

Document frequencies of features extracted from source code, that is, how many documents (repositories, files or functions) contain each tokenized feature.

Example:

from sourced.ml.models import DocumentFrequencies
df = DocumentFrequencies().load(docfreq)
print("Number of tokens:", len(df))

2 models:

id2vec

Source code identifier embeddings, that is, every identifier is represented by a dense vector.

Example:

from sourced.ml.models import Id2Vec
id2vec = Id2Vec().load(id2vec)
print("Number of tokens:", len(id2vec))

2 models:

id_splitter_bilstm

Model that contains source code identifier splitter BiLSTM weights.

Example:

from sourced.ml.models.id_splitter import IdentifierSplitterBiLSTM
id_splitter = IdentifierSplitterBiLSTM().load(id_splitter_bilstm)
id_splitter.split(identifiers)

1 model:

topics

Topic modeling of Git repositories. All tokens are identifiers extracted from repositories and seen as indicators for topics. They are used to infer the topic(s) of repositories.

Example:

from sourced.ml.models import Topics
topics = Topics().load(topics)
print("Number of topics:", len(topics))
print("Number of tokens:", len(topics.tokens))

1 model:

typos_correction

Model that suggests fixes to correct typos.

Example:

from lookout.style.typos.corrector import TyposCorrector
corrector = TyposCorrector().load(typos_correction)
print("Corrector configuration:\n", corrector.dump())

3 models:

About

Machine learning models for MLonCode trained using the source{d} stack

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published