[[
wikihub
]]
Search
⌘K
Explore
People
For Agents
Sign in
Explore
People
For Agents
Sign in
@jemoka / Jemoka Knowledge Base / wiki/concepts/aibridge_iris_variance_worksheet.md
Suggest edit
Cancel
Submit suggestion
Title
Name
Note
--- title: "AIBridge Iris Variance Worksheet" type: concept source: https://www.jemoka.com/posts/kbhaibridge_iris_variance_worksheet/ confidence: high status: active --- SPOILER ALERT for future labs!! Don’t scroll down! We are going to create a copy of the iris dataset with a random variance. import sklearn from sklearn.datasets import load_iris Let’s load the iris dataset: x,y = load_iris(return_X_y=True) Because we need to generate a lot of random data, let’s import random import random Put this in a df import pandas as pd df = pd.DataFrame(x) df 0 1 2 3 0 5.1 3.5 1.4 0.2 1 4.9 3.0 1.4 0.2 2 4.7 3.2 1.3 0.2 3 4.6 3.1 1.5 0.2 4 5.0 3.6 1.4 0.2 .. ... ... ... ... 145 6.7 3.0 5.2 2.3 146 6.3 2.5 5.0 1.9 147 6.5 3.0 5.2 2.0 148 6.2 3.4 5.4 2.3 149 5.9 3.0 5.1 1.8 [150 rows x 4 columns] Let’s make 150 random numbers with pretty low variance: random_ns = [random.uniform(65,65.2) for _ in range(0, 150)] random_series = pd.Series(random_ns) random_series 0 65.127515 1 65.034572 2 65.123271 3 65.043985 4 65.145743 ... 145 65.036410 146 65.157172 147 65.034925 148 65.037373 149 65.042466 Length: 150, dtype: float64 Excellent. Now let’s put the two things together! df["temp"] = random_series df 0 1 2 3 temp 0 5.1 3.5 1.4 0.2 65.127515 1 4.9 3.0 1.4 0.2 65.034572 2 4.7 3.2 1.3 0.2 65.123271 3 4.6 3.1 1.5 0.2 65.043985 4 5.0 3.6 1.4 0.2 65.145743 .. ... ... ... ... ... 145 6.7 3.0 5.2 2.3 65.036410 146 6.3 2.5 5.0 1.9 65.157172 147 6.5 3.0 5.2 2.0 65.034925 148 6.2 3.4 5.4 2.3 65.037373 149 5.9 3.0 5.1 1.8 65.042466 [150 rows x 5 columns] And, while we are at it, let’s make new labels names = pd.Series(["sepal length", "sepal width", "pedal length", "pedal width", "temp"]) df.columns = names df sepal length sepal width pedal length pedal width temp 0 5.1 3.5 1.4 0.2 65.127515 1 4.9 3.0 1.4 0.2 65.034572 2 4.7 3.2 1.3 0.2 65.123271 3 4.6 3.1 1.5 0.2 65.043985 4 5.0 3.6 1.4 0.2 65.145743 .. ... ... ... ... ... 145 6.7 3.0 5.2 2.3 65.036410 146 6.3 2.5 5.0 1.9 65.157172 147 6.5 3.0 5.2 2.0 65.034925 148 6.2 3.4 5.4 2.3 65.037373 149 5.9 3.0 5.1 1.8 65.042466 [150 rows x 5 columns] Excellent. Let’s finally get the flower results. df["species"] = y df sepal length sepal width pedal length pedal width temp species 0 5.1 3.5 1.4 0.2 65.127515 0 1 4.9 3.0 1.4 0.2 65.034572 0 2 4.7 3.2 1.3 0.2 65.123271 0 3 4.6 3.1 1.5 0.2 65.043985 0 4 5.0 3.6 1.4 0.2 65.145743 0 .. ... ... ... ... ... ... 145 6.7 3.0 5.2 2.3 65.036410 2 146 6.3 2.5 5.0 1.9 65.157172 2 147 6.5 3.0 5.2 2.0 65.034925 2 148 6.2 3.4 5.4 2.3 65.037373 2 149 5.9 3.0 5.1 1.8 65.042466 2 [150 rows x 6 columns] And dump it to a CSV. df.to_csv("./iris_variance.csv", index=False) Let’s select for the input data again: X = df.iloc[:,0:5] y = df.iloc[:,5] X sepal length sepal width pedal length pedal width temp 0 5.1 3.5 1.4 0.2 65.127515 1 4.9 3.0 1.4 0.2 65.034572 2 4.7 3.2 1.3 0.2 65.123271 3 4.6 3.1 1.5 0.2 65.043985 4 5.0 3.6 1.4 0.2 65.145743 .. ... ... ... ... ... 145 6.7 3.0 5.2 2.3 65.036410 146 6.3 2.5 5.0 1.9 65.157172 147 6.5 3.0 5.2 2.0 65.034925 148 6.2 3.4 5.4 2.3 65.037373 149 5.9 3.0 5.1 1.8 65.042466 [150 rows x 5 columns] And use the variance threshold tool: from sklearn.feature_selection import VarianceThreshold sel = VarianceThreshold(0.1) sel.fit_transform(X) 5.1 3.5 1.4 0.2 4.9 3 1.4 0.2 4.7 3.2 1.3 0.2 4.6 3.1 1.5 0.2 5 3.6 1.4 0.2 5.4 3.9 1.7 0.4 4.6 3.4 1.4 0.3 … As we expected. And let’s use the select k best tool: from sklearn.feature_selection import SelectKBest, chi2 sel = SelectKBest(chi2, k=4) res = sel.fit_transform(X, y) res 5.1 3.5 1.4 0.2 4.9 3 1.4 0.2 4.7 3.2 1.3 0.2 4.6 3.1 1.5 0.2 5 3.6 1.4 0.2 5.4 3.9 1.7 0.4 4.6 3.4 1.4 0.3 5 3.4 1.5 0.2 … Also, as we expected. Got rid of temp.