Text Classification

Code

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

texts = [
    "I love this movie", "great film", "awesome and fun",
    "terrible acting", "boring and slow", "I hated every minute",
    "wonderful story", "bad plot",
]
labels = ["pos", "pos", "pos", "neg", "neg", "neg", "pos", "neg"]

X_tr, X_te, y_tr, y_te = train_test_split(texts, labels, test_size=0.25,
                                          random_state=42)

pipe = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=1)),
    ("clf", LogisticRegression(C=1.0, max_iter=1000)),
])
pipe.fit(X_tr, y_tr)
pred = pipe.predict(X_te)
print(classification_report(y_te, pred))

# Predict new text
print(pipe.predict(["an amazing experience", "a dull waste of time"]))

Explanation

A Pipeline that vectorizes text with TF-IDF and trains a logistic regression is a robust baseline for text classification. ngram_range=(1, 2) lets the model use bigrams such as 'not good' that flip sentiment. The same predict interface used at evaluation works on new raw strings, making deployment straightforward.

More NLP Snippets