Hyperparameter Tuning and Experiment Tracking with Weights & Biases

In this article, I will discuss how I used Weights & Biases (W&B) to track various metrics and plots during a machine learning experiment. W&B provides an interface for tracking experiments, logging metrics, visualizing results, and optimizing model performance.

Code

GitHub Repository:

https://github.com/masaki9/ml_experiments_with_wandb

Setting Up the Sweep Configuration

To start hyperparameter tuning with W&B, I defined a sweep configuration in a YAML file. The configuration specifies the hyperparameters to explore, the search method (e.g. grid search), and the metric (e.g. F1) to optimize.

method: grid
metric:
  name: 'F1 Score (Weighted)'
  goal: maximize
parameters:
  C:
    values: [0.3, 0.5, 0.7, 1.0, 1.5]
  coef0:
    values: [0.3, 0.5, 0.7, 1.0, 1.5]
  degree:
    values: [1, 2, 3]

I use a grid search method in this setup to explore hyperparameters C, coef0, and degree for a Support Vector Machine (SVM) model with a polynomial kernel. This sweep aims to maximize the weighted F1 score metric appropriate for imbalanced data.

Code for the Sweep

The script (main.py) initializes the W&B sweep, sets up the SVM model pipeline, and logs various metrics and plots during training and evaluation. The following is an excerpt from the script:

# Main function for running the sweep
def main():
    wandb.init()  # Initialize a wandb run

    # Prepare the data
    aids_clinical_trials_group_study_175 = fetch_ucirepo(id=890)
    X = aids_clinical_trials_group_study_175.data.features
    y = aids_clinical_trials_group_study_175.data.targets.squeeze()
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

    # Hyperparameters
    C = wandb.config.C
    coef0 = wandb.config.coef0
    degree = wandb.config.degree

    # Set up the SVM pipeline
    svm_pipe = make_pipeline(StandardScaler(), SVC(kernel='poly', C=C, coef0=coef0, degree=degree))
    svm_pipe.fit(X_train, y_train)
    y_pred = svm_pipe.predict(X_test)

    # Log the learning curve
    plot_learning_curve(svm_pipe, X_train, y_train, cv=cv, scoring='f1_weighted', title='SVM Learning Curve (Polynomial Kernel)', ylabel='F1 Score (Weighted)', ylim=(0.5, 1.0))
    wandb.log({"learning_curve": wandb.Image(plt)})
    plt.close()

    # Log the classification report as a table
    class_report_dict = classification_report(y_test, y_pred, output_dict=True)
    columns = ["Class", "Precision", "Recall", "F1-Score", "Support"]
    table_data = [[label, metrics['precision'], metrics['recall'], metrics['f1-score'], metrics['support']] for label, metrics in class_report_dict.items() if label != "accuracy"]
    class_report_table = wandb.Table(columns=columns, data=table_data)
    wandb.run.summary["classification_report_table"] = class_report_table

    # Calculate and log the weighted F1 score
    weighted_f1_score = f1_score(y_test, y_pred, average='weighted')
    wandb.run.summary["F1 Score (Weighted)"] = weighted_f1_score

    wandb.finish()  # Finish the wandb run

# Load the sweep configuration and run the sweep
with open('sweep_config.yml') as file:
    sweep_config = yaml.safe_load(file)
    sweep_id = wandb.sweep(sweep=sweep_config, project='svm-poly-kernel')
    wandb.agent(sweep_id, function=main)

Running the Sweep

The sweep is initiated with the following lines of code:

sweep_id = wandb.sweep(sweep=sweep_config, project='svm-poly-kernel')
wandb.agent(sweep_id, function=main)

W&B runs the sweep with different hyperparameter combinations defined in the YAML file. Each run logs the metrics and plots to the W&B dashboard for easy comparison.

Visualizing Results in the W&B Dashboard

The W&B dashboard provides an interactive visualization of the sweep results. Here are some highlights:

1. Parameter Importance: This plots shows the impact of each hyperparameter (degree, C, coef0) on the model’s performance (weighted F1 Score).
2. Parallel Coordinates: This plot shows how different hyperparameter combinations affect the model’s performance.
3. Runs Table This table displays a sortable list of each run and allows for easy comparison between the metrics and hyperparameters used.
4. Run Summary: Each run’s summary can include the metrics and plots such as a learning curve and confusin matrix, which allows for a review of how the model performed with the set of parameters.

Conclusion

W&B provides a comprehensive and interactive way to visualize and optimize model performance. The W&B dashboard, along with parallel coordinates plots and run summaries, makes it easy to identify the best hyperparameter combinations and understand their effects.