我們使用一個(gè)簡(jiǎn)單的k-means算法來(lái)演示如何進(jìn)行聚類。聚類可以幫助發(fā)現(xiàn)數(shù)據(jù)中有價(jià)值的隱藏分組。數(shù)據(jù)集是在 Get_embeddings_from_dataset Notebook中創(chuàng)建的。

# 導(dǎo)入必要的庫(kù)
import numpy as np
import pandas as pd
from ast import literal_eval# 數(shù)據(jù)文件路徑
datafile_path = "./data/fine_food_reviews_with_embeddings_1k.csv"# 讀取csv文件為DataFrame格式
df = pd.read_csv(datafile_path)# 將embedding列中的字符串轉(zhuǎn)換為numpy數(shù)組
df["embedding"] = df.embedding.apply(literal_eval).apply(np.array)# 將所有的embedding數(shù)組按行堆疊成一個(gè)矩陣
matrix = np.vstack(df.embedding.values)# 輸出矩陣的形狀
matrix.shape

(1000, 1536)

1. 使用K-means算法找到聚類

我們展示了K-means的最簡(jiǎn)單用法。您可以選擇最適合您用例的聚類數(shù)量。

# 導(dǎo)入KMeans聚類算法
from sklearn.cluster import KMeans# 設(shè)置聚類數(shù)目
n_clusters = 4# 初始化KMeans算法，設(shè)置聚類數(shù)目、初始化方法和隨機(jī)種子
kmeans = KMeans(n_clusters=n_clusters, init="k-means++", random_state=42)# 使用KMeans算法對(duì)數(shù)據(jù)進(jìn)行聚類
kmeans.fit(matrix)# 獲取聚類標(biāo)簽
labels = kmeans.labels_# 將聚類標(biāo)簽添加到數(shù)據(jù)框中
df["Cluster"] = labels# 按照聚類標(biāo)簽對(duì)數(shù)據(jù)框進(jìn)行分組，計(jì)算每個(gè)聚類的平均分?jǐn)?shù)，并按照平均分?jǐn)?shù)排序
df.groupby("Cluster").Score.mean().sort_values()

/Users/ted/.virtualenvs/openai/lib/python3.9/site-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warningwarnings.warn(Cluster
0    4.105691
1    4.191176
2    4.215613
3    4.306590
Name: Score, dtype: float64

# 導(dǎo)入必要的庫(kù)
from sklearn.manifold import TSNE
import matplotlib
import matplotlib.pyplot as plt# 初始化t-SNE模型，設(shè)置參數(shù)
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init="random", learning_rate=200)# 使用t-SNE模型對(duì)數(shù)據(jù)進(jìn)行降維
vis_dims2 = tsne.fit_transform(matrix)# 提取降維后的數(shù)據(jù)的x和y坐標(biāo)
x = [x for x, y in vis_dims2]
y = [y for x, y in vis_dims2]# 針對(duì)每個(gè)類別，繪制散點(diǎn)圖，并標(biāo)記類別的平均值
for category, color in enumerate(["purple", "green", "red", "blue"]):# 提取屬于當(dāng)前類別的數(shù)據(jù)的x和y坐標(biāo)xs = np.array(x)[df.Cluster == category]ys = np.array(y)[df.Cluster == category]# 繪制散點(diǎn)圖plt.scatter(xs, ys, color=color, alpha=0.3)# 計(jì)算當(dāng)前類別的平均值avg_x = xs.mean()avg_y = ys.mean()# 標(biāo)記平均值plt.scatter(avg_x, avg_y, marker="x", color=color, s=100)# 設(shè)置圖表標(biāo)題
plt.title("Clusters identified visualized in language 2d using t-SNE")

Text(0.5, 1.0, 'Clusters identified visualized in language 2d using t-SNE')

在二維投影中對(duì)聚類進(jìn)行可視化。在這次運(yùn)行中，綠色聚類（#1）似乎與其他聚類非常不同。讓我們看一下每個(gè)聚類的幾個(gè)樣本。

2. 聚類中的文本樣本和聚類的命名讓我們展示每個(gè)聚類中的隨機(jī)樣本。

我們將使用text-davinci-003來(lái)為聚類命名，基于從該聚類中隨機(jī)抽取的5個(gè)評(píng)論樣本。

# 導(dǎo)入openai模塊import openai# 每個(gè)聚類組中的評(píng)論數(shù)量
rev_per_cluster = 5# 遍歷每個(gè)聚類組
for i in range(n_clusters):# 輸出聚類組的主題print(f"Cluster {i} Theme:", end=" ")# 選取屬于該聚類組的評(píng)論，并將它們合并成一個(gè)字符串reviews = "\n".join(df[df.Cluster == i].combined.str.replace("Title: ", "").str.replace("\n\nContent: ", ":  ").sample(rev_per_cluster, random_state=42).values)# 使用openai模塊對(duì)選取的評(píng)論進(jìn)行主題分析response = openai.Completion.create(engine="text-davinci-003",prompt=f'What do the following customer reviews have in common?\n\nCustomer reviews:\n"""\n{reviews}\n"""\n\nTheme:',temperature=0,max_tokens=64,top_p=1,frequency_penalty=0,presence_penalty=0,)# 輸出主題分析結(jié)果print(response["choices"][0]["text"].replace("\n", ""))# 選取屬于該聚類組的樣本行，并輸出它們的得分、摘要和文本內(nèi)容sample_cluster_rows = df[df.Cluster == i].sample(rev_per_cluster, random_state=42)for j in range(rev_per_cluster):print(sample_cluster_rows.Score.values[j], end=", ")print(sample_cluster_rows.Summary.values[j], end=":   ")print(sample_cluster_rows.Text.str[:70].values[j])# 輸出分隔符print("-" * 100)

Cluster 0 Theme:  All of the reviews are positive and the customers are satisfied with the product they purchased.
5, Loved these gluten free healthy bars, saved $$ ordering on Amazon:   These Kind Bars are so good and healthy & gluten free.  My daughter ca
1, Should advertise coconut as an ingredient more prominently:   First, these should be called Mac - Coconut bars, as Coconut is the #2
5, very good!!:   just like the runts<br />great flavor, def worth getting<br />I even o
5, Excellent product:   After scouring every store in town for orange peels and not finding an
5, delicious:   Gummi Frogs have been my favourite candy that I have ever tried. of co
----------------------------------------------------------------------------------------------------
Cluster 1 Theme:  All of the reviews are about pet food.
2, Messy and apparently undelicious:   My cat is not a huge fan. Sure, she'll lap up the gravy, but leaves th
4, The cats like it:   My 7 cats like this food but it is a little yucky for the human. Piece
5, cant get enough of it!!!:   Our lil shih tzu puppy cannot get enough of it. Everytime she sees the
1, Food Caused Illness:   I switched my cats over from the Blue Buffalo Wildnerness Food to this
5, My furbabies LOVE these!:   Shake the container and they come running. Even my boy cat, who isn't 
----------------------------------------------------------------------------------------------------
Cluster 2 Theme:  All of the reviews are positive and express satisfaction with the product.
5, Fog Chaser Coffee:   This coffee has a full body and a rich taste. The price is far below t
5, Excellent taste:   This is to me a great coffee, once you try it you will enjoy it, this 
4, Good, but not Wolfgang Puck good:   Honestly, I have to admit that I expected a little better. That's not 
5, Just My Kind of Coffee:   Coffee Masters Hazelnut coffee used to be carried in a local coffee/pa
5, Rodeo Drive is Crazy Good Coffee!:   Rodeo Drive is my absolute favorite and I'm ready to order more!  That
----------------------------------------------------------------------------------------------------
Cluster 3 Theme:  All of the reviews are about food or drink products.
5, Wonderful alternative to soda pop:   This is a wonderful alternative to soda pop.  It's carbonated for thos
5, So convenient, for so little!:   I needed two vanilla beans for the Love Goddess cake that my husbands 
2, bot very cheesy:   Got this about a month ago.first of all it smells horrible...it tastes
5, Delicious!:   I am not a huge beer lover.  I do enjoy an occasional Blue Moon (all o
3, Just ok:   I bought this brand because it was all they had at Ranch 99 near us. I
----------------------------------------------------------------------------------------------------

重要的是要注意，聚類不一定與您打算使用它們的目的完全匹配。更多的聚類將關(guān)注更具體的模式，而較少的聚類通常會(huì)關(guān)注數(shù)據(jù)中最大的差異。