dojo_ds package

Custom Functions the Data Science Program __author__ = James Irving, Brenda Hungerford

dojo_ds.show_code(function)[source]

Uses the inspect module to retrieve the source code for a function. Displays the code as Python-syntax Markdown code.

Note: Python highlighting may not work correctly on some editors.

Parameters: function (callable): The function for which to display the source code.

Returns: None

Submodules

dojo_ds.cli module

dojo_ds.data_enrichment module

dojo_ds.data_enrichment.find_outliers_IQR(data, verbose=True)[source]

Find outliers based on IQR-rule (outliers are either 1.5 x IQR below 25% quantile and 1.5xIQR above 75% quantile).

Parameters:

  • data (Series) – Pandas Series containing the data.

  • verbose (bool, optional) – If True, print summary information about outliers. Defaults to True.

Returns:

Boolean index for input data, where True indicates an outlier.

Return type:

Series

dojo_ds.data_enrichment.find_outliers_Z(data, verbose=True)[source]

Find outliers based on Z-score rule (outliers have an absolute z-score that is >3)

Parameters:

  • data (Series) – Pandas Series

  • verbose (bool, optional) – Print summary info about outliers. Defaults to True.

Returns:

Boolean index for input data, where True = Outlier

Return type:

Series

dojo_ds.data_enrichment.remove_outliers(df_, method='iqr', subset=None, verbose=2)[source]

Returns a copy of the input dataframe with outliers removed from selected columns using the specified method.

Parameters:

  • df (DataFrame) – The input dataframe to copy and remove outliers from.

  • method (str) – The method of outlier removal. Options are ‘iqr’ or ‘z’/’zscore’. Default is ‘iqr’.

  • subset (list or None) – A list of column names to remove outliers from. If None, all numeric columns are used. Default is None.

  • verbose (bool, int) – If verbose==1, print only the overall summary. If verbose==2, print the detailed summary. Default is 2.

Returns:

A copy of the input dataframe with outliers removed.

Return type:

DataFrame

Raises:

Exception – If the method is not ‘iqr’ or ‘z’.

Examples

>>> df = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 20, 30, 40, 50]})
>>> remove_outliers(df, method='iqr', subset=['A'], verbose=2)
Returns a dataframe with outliers removed from column 'A' using the IQR rule.

dojo_ds.deploy module

dojo_ds.deploy.load_filepath_config(config_fpath='config/filepaths.json', verbose=True)[source]

Loads the filepaths configuration from a JSON file.

Parameters: - config_fpath (str): The filepath of the JSON configuration file. Default is ‘config/filepaths.json’. - verbose (bool): Whether to print the loaded filepaths. Default is True.

Returns: - dict: A dictionary containing the loaded filepaths.

dojo_ds.deploy.save_filepath_config(FPATHS, overwrite=True, output_fpath='config/filepaths.json', verbose=False)[source]

Save the filepaths to a JSON file.

Parameters: FPATHS (dict): A dictionary containing the filepaths. overwrite (bool): Whether to overwrite the existing file if it already exists. Default is True. output_fpath (str): The output filepath to save the JSON file. Default is ‘config/filepaths.json’. verbose (bool): Whether to print the saved filepaths. Default is False.

Returns: dict: The dictionary containing the filepaths.

Raises: Exception: If the output file already exists and overwrite is set to False.

dojo_ds.eda module

dojo_ds.eda.annotate_regplot_equation(ax)[source]

Annotates a regression plot with the equation of the regression line.

Parameters: ax (matplotlib.axes.Axes): The axes object containing the regression plot.

Adapted from Source: https://www.statology.org/seaborn-regplot-equation/

Example Use: >> fig, ax = plot_numeric_vs_target(df, x=”Living Area Sqft”) >> annotate_regplot_equation(ax)

dojo_ds.eda.explore_categorical(df, x, fillna=True, placeholder='MISSING', figsize=(6, 4), order=None, show=True)[source]

Plots a seaborn countplot of for x column and prints information on: - the # and % of null values - number of unique values - the most frequent category and how much of the feature is this category (%) - A warning message if the feature is quasi-constant or constant feature

(if more than 99% of feature is a single value)

Parameters:

  • df (Frame) – DataFrame that contains column x

  • x (str) – a column name

  • fillna (bool, optional) – if True, fillna with the placeholder. Defaults to True.

  • placeholder (str, optional) – Value used to fillna if fillna is True. Defaults to ‘MISSING’.

  • figsize (tuple, optional) – Figure size (width, height). Defaults to (6,4).

  • order (list, optional) –

    List of categories to include in graph, in the specified order. Defaults to None. Note: any category not in the order list will not be shown on the graph.

    If a category is included in the order list that isn’t in the data, it will be added as an empty bar categories can be removed from the visuals

Returns:

Matplotlib Figure ax: Matplotlib Axes

Return type:

fig

dojo_ds.eda.explore_numeric(df, x, figsize=(6, 5), show=True)[source]
Plots a Seaborn histplot on the top subplot and a horizontal boxplot on he bottom.

Additionally, prints information on: - the # and % of null values - number of unique values - the most frequent value and how often frequent it is (%) - A warning message if the feature is quasi-constant or constant feature

(if more than 99% of feature is a single value)

Args:

df (Frame): DataFrame that contains column x x (str): a column name fillna (bool, optional): if True, fillna with the placeholder. Defaults to True. placeholder (str, optional): Value used to fillna if fillna is True. Defaults to ‘MISSING’. figsize (tuple, optional): Figure size (width, height). Defaults to (6,5). order (list, optional): List of categories to include in graph, in the specified order. Defaults to None.

Note: any category not in the order list will not be shown on the graph.

If a category is included in the order list that isn’t in the data, it will be added as an empty bar categories can be removed from the visuals

Returns:

fig: Matplotlib Figure ax: Matplotlib Axes

Source: https://login.codingdojo.com/m/606/13765/117605

dojo_ds.eda.plot_categorical_vs_target(df, x, y, target_type='reg', figsize=(6, 4), fillna=True, placeholder='MISSING', order=None, show=True)[source]

Updated Version of the function which accepts either numeric or categorical targets. Adapted from Source: https://login.codingdojo.com/m/606/13765/117606 Plots a combination seaborn barplot (without error bars) and a stripplot.

Args:

df (Frame): DataFrame containing data to plot. x (str): Column to use as the x-axis (categories) y (str, optional): Target column to plot on the y-axis. Defaults to ‘SalePrice’. fillna (bool, optional): if True, fillna with the placeholder. Defaults to True. placeholder (str, optional): Value used to fillna if fillna is True. Defaults to ‘MISSING’. figsize (tuple, optional): Figure size (width, height). Defaults to (6,4). order (list, optional): List of categories to include in graph, in the specified order. Defaults to None.

Note: any category not in the order list will not be shown on the graph.

If a category is included in the order list that isn’t in the data, it will be added as an empty bar categories can be removed from the visuals

Returns:

fig: Matplotlib Figure ax: Matplotlib Axes

dojo_ds.eda.plot_correlation(df, cmap='coolwarm', cols=None)[source]

Plots a correlation matrix heatmap for the given DataFrame.

Parameters: df (DataFrame): The input DataFrame. cmap (str, optional): The color map to use for the heatmap. Defaults to ‘coolwarm’. cols (list, optional): The columns to include in the correlation matrix. If None, all columns are included.

Returns: fig (Figure): The matplotlib Figure object containing the correlation matrix heatmap.

dojo_ds.eda.plot_numeric_vs_target(df, x, y, figsize=(6, 4), target_type='reg', errorbar='ci', estimator='mean', order=None, show=True, **kwargs)[source]

UPDATED FUNCTION WITH OPTION FOR WHICH TYPE OF TARGET Source: https://login.codingdojo.com/m/606/13765/117605 Plots a seaborn regplot, with an optional formula annotation.

Also calculates correlation and displays Pearson’s r in the title.

Args:

df (Frame): DataFrame with data. x (str): Numeric column name. y (str, optional): Numeric target column name. Defaults to ‘SalePrice’. figsize (tuple, optional): Figure size. Defaults to (6,4). annotate (bool, optional): Whether to annotate regplot equation. Defaults to False.

Returns:

fig: Matplotlib Figure ax: Matplotlib Axes

dojo_ds.eda.summarize_df(df_)[source]

Source: Insights for Stakeholder Lesson - https://login.codingdojo.com/m/0/13079/91969 Example Usage: >> df = pd.read_csv(filename) >> summary = summarize_df(df) >> summary

dojo_ds.evaluate module

dojo_ds.evaluate.classification_metrics(y_true, y_pred, label='', output_dict=False, figsize=(8, 4), normalize='true', cmap='Blues', colorbar=False, values_format='.2f', target_names=None, return_fig=True)[source]

Calculate classification metrics from predictions and display Confusion matrix.

Parameters:

  • y_true (Series/array) – True target values.

  • y_pred (Series/array) – Predicted target values.

  • label (str, optional) – Label for printed header. Defaults to ‘’.

  • output_dict (bool, optional) – Return the results of classification_report as a dict. Defaults to False.

  • figsize (tuple, optional) – figsize for confusion matrix subplots. Defaults to (8,4).

  • normalize (str, optional) – Arg for sklearn’s ConfusionMatrixDisplay. Defaults to ‘true’ (conf mat values normalized to true class).

  • cmap (str, optional) – Colormap for the ConfusionMatrixDisplay. Defaults to ‘Blues’.

  • colorbar (bool, optional) – Arg for ConfusionMatrixDisplay: include colorbar or not. Defaults to False.

  • values_format (str, optional) – Format values on confusion matrix. Defaults to “.2f”.

  • target_names (array, optional) – Text labels for the integer-encoded target. Passed in numeric order [label for “0”, label for “1”, etc.].

  • return_fig (bool, optional) – To get matplotlib figure for confusion matrix, set output_dict to False and set return_fig to True.

Returns:

Dictionary from classification_report. Only returned if output_dict=True. fig: Matplotlib figure with confusion matrix. Only returned if output_dict=False and return_fig=True.

Return type:

dict

Note

This is a modified version of the classification metrics function from Intro to Machine Learning. Updates:

  • Reversed raw counts confusion matrix cmap (so darker==more).

  • Added arg for normalized confusion matrix values_format.

dojo_ds.evaluate.convert_y_to_sklearn_classes(y, verbose=False)[source]

Helper function to convert neural network outputs to class labels.

Parameters:

  • y (array/Series) – Predictions to convert to classes.

  • verbose (bool, optional) – Print which preprocessing approach is used. Defaults to False.

Returns:

Target as 1D class labels

Return type:

array

dojo_ds.evaluate.evaluate_classification(model, X_train=None, y_train=None, X_test=None, y_test=None, figsize=(6, 4), normalize='true', output_dict=False, cmap_train='Blues', cmap_test='Reds', colorbar=False, values_format='.2f', target_names=None, return_fig=False)[source]

Evalutes an sklearn-compatible classification model on training and test data. For each data split, return the classification report and confusion matrix display.

Parameters:

  • model (sklearn estimator) – Classification model to evaluate.

  • X_train (Frame/Array, optional) – Training data. Defaults to None.

  • y_train (Series/Array, optional) – Training labels. Defaults to None.

  • X_test (Frame/Array, optional) – Test data. Defaults to None.

  • y_test (Series/Array, optional) – Test labels. Defaults to None.

  • figsize (tuple, optional) – figsize for confusion matrix subplots. Defaults to (6,4).

  • normalize (str, optional) – arg for sklearn’s ConfusionMatrixDisplay. Defaults to ‘true’ (conf mat values normalized to true class).

  • output_dict (bool, optional) – Return the results of classification_report as a dict. Defaults to False. Defaults to False.

  • cmap_train (str, optional) – Colormap for the ConfusionMatrixDisplay for training data. Defaults to ‘Blues’.

  • cmap_test (str, optional) – Colormap for the ConfusionMatrixDisplay for test data. Defaults to “Reds”.

  • colorbar (bool, optional) – Arg for ConfusionMatrixDispaly: include colorbar or not. Defaults to False.

  • values_format (str, optional) – Format values on confusion matrix. Defaults to “.2f”.

  • target_names (array, optional) – Text labels for the integer-encoded target. Passed in numeric order [label for “0”, label for “1”, etc.]

  • return_fig – Whether the matplotlib figure for confusion matrix is returned. Defaults to False. Note: Must set outout_dict to False and set return_fig to True to get figure returned.

dojo_ds.evaluate.evaluate_classification_network(model, X_train=None, y_train=None, X_test=None, y_test=None, history=None, history_figsize=(6, 6), figsize=(6, 4), normalize='true', output_dict=False, cmap_train='Blues', cmap_test='Reds', values_format='.2f', colorbar=False, target_names=None, return_fig=False)[source]

Evaluates a neural network classification task using either separate X and y arrays or a tensorflow Dataset

Parameters:

  • model (sklearn-compatible classifier) – Model to evaluate.

  • X_train (array or tf.data.Dataset, optional) – Training data. Defaults to None.

  • y_train (array, or None if X_train is a tf Dataset, optional) – Training labels (if not using a tf dataset). Defaults to None.

  • X_test (array or tf.data.Dataset, optional) – Test data. Defaults to None.

  • y_test (array, or None if X_test is a tf Dataset, optional) – Test labels (if not using a tf Dataset). Defaults to None.

  • history (tensorflow history object, optional) – History object from model training. Defaults to None.

  • history_figsize (tuple, optional) – Total figure size for plot_history. Defaults to (6,8).

  • figsize (tuple, optional) – figsize for confusion matrix subplots. Defaults to (6,4).

  • normalize (str, optional) – arg for sklearn’s ConfusionMatrixDisplay. Defaults to ‘true’ (conf mat values normalized to true class).

  • output_dict (bool, optional) – Return the results of classification_report as a dict. Defaults to False. Defaults to False.

  • cmap_train (str, optional) – Colormap for the ConfusionMatrixDisplay for training data. Defaults to ‘Blues’.

  • cmap_test (str, optional) – Colormap for the ConfusionMatrixDisplay for test data. Defaults to “Reds”.

  • colorbar (bool, optional) – Arg for ConfusionMatrixDispaly: include colorbar or not. Defaults to False.

  • values_format (str, optional) – Format values on confusion matrix. Defaults to “.2f”.

  • target_names (array, optional) – Text labels for the integer-encoded target. Passed in numeric order [label for “0”, label for “1”, etc.]

  • return_fig – Whether the matplotlib figure for confusion matrix is returned. Defaults to False. Note: Must set outout_dict to False and set return_fig to True to get figure returned.

dojo_ds.evaluate.evaluate_ols(result, X_train_df, y_train, show_summary=True)[source]

Plots a Q-Q Plot and residual plot for a statsmodels OLS regression, with option to display summary.

Parameters:

  • result (statsmodels RegressionResultsWrapper) – The result object obtained from fitting the OLS model.

  • X_train_df (pandas DataFrame) – The training data features.

  • y_train (pandas Series) – The training data labels.

  • show_summary (bool, optional) – Whether to display the summary of the regression model. Defaults to True.

dojo_ds.evaluate.evaluate_regression(reg, X_train, y_train, X_test, y_test, verbose=True, output_frame=False)[source]

Evalutes an sklearn-compatible regression model on training and test data. For each data split, return the classification report and confusion matrix display.

Parameters:

  • reg (sklearn estimator) – Regression model to evaluate.

  • X_train (Frame/Array, optional) – Training data. Defaults to None.

  • y_train (Series/Array, optional) – Training labels. Defaults to None.

  • X_test (Frame/Array, optional) – Test data. Defaults to None.

  • y_test (Series/Array, optional) – Test labels. Defaults to None.

  • verbose (bool, optional) – Controls printing of results. Defaults to True.

  • output_dict (bool, optional) – Return results in a dictionary. Defaults to False. (Note one of either verbose or output_dict should be set to True)

Returns:

Dictionary of results with keys: ‘Label’,’MAE’,’MSE’, ‘RMSE’, ‘MAPE’,’R^2’. Only returned if output_dict==True.

Return type:

dict

dojo_ds.evaluate.get_true_pred_labels(model, ds)[source]

Gets the true labels and predicted probabilities from a Tensorflow model and Dataset object.

Parameters:

  • model (Tensorflow/Keras model) – The model to get predictions from.

  • ds (tensorflow.data.Dataset) – The dataset to iterate through.

Returns:

A tuple containing the true labels and predicted probabilities.

Return type:

tuple

dojo_ds.evaluate.plot_history(history, figsize=(6, 8), return_fig=False)[source]

Plots the training and validation curves for all metrics in a Tensorflow History object.

Parameters:

  • history (Tensorflow History) – History output from training a neural network.

  • figsize (tuple, optional) – Total figure size. Defaults to (6,8).

  • return_fig (boolean, optional) – If true, return figure instead of displaying it with plt.show()

Returns:

If return_fig is True, returns the figure object. Otherwise, displays the figure using plt.show().

Return type:

None or matplotlib.figure.Figure

dojo_ds.evaluate.plot_residuals(model, X_test_df, y_test, figsize=(12, 5))[source]

Plots a Q-Q Plot and residual plot for a regression model.

Parameters:

  • model (regression model) – The regression model that supports .predict.

  • X_test_df (DataFrame) – The test data.

  • y_test (array-like) – The test labels.

  • figsize (tuple, optional) – The figsize for the regression plots. Defaults to (12,5).

dojo_ds.evaluate.regression_metrics(y_true, y_pred, label='', verbose=True, output_dict=False)[source]

Calculate MEA, MSE, RMSE, R-Squared and MAPE using the true and predicted labels.

Parameters:

  • y_true (Series/array) – True target values.

  • y_pred (Series/array) – Predicted target values.

  • label (str, optional) – Label to display in results header. Defaults to ‘’.

  • verbose (bool, optional) – Controls printing of results. Defaults to True.

  • output_dict (bool, optional) – Return results in a dictionary. Defaults to False. (Note one of either verbose or output_dict should be set to True)

Returns:

Dictionary of results with keys: ‘Label’, ‘MAE’, ‘MSE’, ‘RMSE’, ‘MAPE’, ‘R^2’.

Only returned if output_dict is True.

Return type:

dict

dojo_ds.imports module

Standard Imports module: lazy importing of essential packages

Example Use: >> from dojo_ds.standard_imports import *

dojo_ds.insights module

dojo_ds.insights.annotate_hbars(ax, ha='left', va='center', size=12, xytext=(4, 0), textcoords='offset points')[source]

Annotates horizontal bars on a matplotlib Axes object.

Parameters: - ax (matplotlib.axes.Axes): The Axes object to annotate. - ha (str): The horizontal alignment of the annotation text. Default is ‘left’. - va (str): The vertical alignment of the annotation text. Default is ‘center’. - size (int): The font size of the annotation text. Default is 12. - xytext (tuple): The offset of the annotation text from the annotated point. Default is (4, 0). - textcoords (str): The coordinate system used for xytext. Default is ‘offset points’.

dojo_ds.insights.get_coefficients(reg, name='Coefficients')[source]

Save a model’s .coef_ and .intercept_ as a Pandas Series

dojo_ds.insights.get_coeffs_linreg(lin_reg, feature_names=None, sort=True, ascending=True, name='LinearRegression Coefficients')[source]

Get the coefficients of a linear regression model.

Parameters: - lin_reg: The trained linear regression model. - feature_names: Optional. The names of the features used in the model. If not provided, it will use the feature names from the model. - sort: Optional. Whether to sort the coefficients by value. Default is True. - ascending: Optional. Whether to sort the coefficients in ascending order. Default is True. - name: Optional. The name of the coefficients series. Default is ‘LinearRegression Coefficients’.

Returns: - coeffs: A pandas Series containing the coefficients of the linear regression model.

dojo_ds.insights.get_coeffs_logreg(logreg, feature_names=None, sort=True, ascending=True, name='LogReg Coefficients', class_index=0, include_intercept=True, as_odds=False)[source]

Get the coefficients of a logistic regression model.

Parameters: logreg (object): The logistic regression model. feature_names (list, optional): List of feature names. If None, it uses the feature names from the model. sort (bool, optional): Whether to sort the coefficients. Default is True. ascending (bool, optional): Whether to sort the coefficients in ascending order. Default is True. name (str, optional): Name of the coefficients. Default is ‘LogReg Coefficients’. class_index (int, optional): Index of the class for which to get the coefficients. Default is 0. include_intercept (bool, optional): Whether to include the intercept in the coefficients. Default is True. as_odds (bool, optional): Whether to exponentiate the coefficients to obtain odds ratios. Default is False.

Returns: pd.Series: Series containing the coefficients.

dojo_ds.insights.get_color_dict(importances, color_rest='#006ba4', color_top='green', top_n=7)[source]

Returns a dictionary mapping feature names to colors based on their importances.

Parameters: importances (pd.Series): A pandas Series containing feature importances. color_rest (str, optional): The color code for non-highlighted features. Defaults to ‘#006ba4’. color_top (str, optional): The color code for highlighted features. Defaults to ‘green’. top_n (int, optional): The number of top features to highlight. Defaults to 7.

Returns: dict: A dictionary mapping feature names to colors.

dojo_ds.insights.get_colors_gt_lt(coeffs, threshold=1, color_lt='darkred', color_gt='forestgreen', color_else='gray')[source]

Creates a dictionary of features and their corresponding colors based on whether the value is greater than or less than the threshold.

Parameters: coeffs (pandas.DataFrame): The coefficients dataframe. threshold (float): The threshold value. Default is 1. color_lt (str): The color for values less than the threshold. Default is ‘darkred’. color_gt (str): The color for values greater than the threshold. Default is ‘forestgreen’. color_else (str): The color for values equal to the threshold. Default is ‘gray’.

Returns: dict: A dictionary mapping features to their respective colors.

dojo_ds.insights.get_importances(model, feature_names=None, name='Feature Importance', sort=False, ascending=True)[source]

Extract the feature importances for a given model.

Parameters: model (object): The trained model for which feature importances are calculated. feature_names (list, optional): List of feature names. If not provided, it will be extracted from the model. name (str, optional): Name of the feature importances series. Default is ‘Feature Importance’. sort (bool, optional): Whether to sort the importances in ascending order. Default is False. ascending (bool, optional): Whether to sort the importances in ascending order. Default is True.

Returns: importances (pd.Series): Series containing the feature importances.

dojo_ds.insights.plot_coefficients(coeffs, figsize=(6, 5), title='Regression Coefficients', intercept=True, intercept_name='Intercept', sort_values=True, ascending=True)[source]
dojo_ds.insights.plot_coeffs(coeffs, top_n=None, figsize=(4, 5), intercept=False, intercept_name='intercept', annotate=False, ha='left', va='center', size=12, xytext=(4, 0), textcoords='offset points')[source]

Plots the top_n coefficients from a Series, with optional annotations.

Parameters: coeffs (pd.Series): The coefficients to be plotted. top_n (int, optional): The number of top coefficients to plot. If None, all coefficients will be plotted. Default is None. figsize (tuple, optional): The size of the figure. Default is (4, 5). intercept (bool, optional): Whether to include the intercept coefficient in the plot. Default is False. intercept_name (str, optional): The name of the intercept coefficient. Default is “intercept”. annotate (bool, optional): Whether to annotate the coefficients on the plot. Default is False. ha (str, optional): The horizontal alignment of the annotations. Default is ‘left’. va (str, optional): The vertical alignment of the annotations. Default is ‘center’. size (int, optional): The font size of the annotations. Default is 12. xytext (tuple, optional): The offset of the annotations from the data points. Default is (4, 0). textcoords (str, optional): The coordinate system used for the annotations. Default is ‘offset points’.

Returns: matplotlib.axes.Axes: The plot of the coefficients.

dojo_ds.insights.plot_coeffs_color(coeffs, top_n=None, figsize=(8, 6), legend_loc='best', threshold=None, color_lt='darkred', color_gt='forestgreen', color_else='gray', label_thresh='Equally Likely', label_gt='More Likely', label_lt='Less Likely', plot_kws={})[source]

Plots series of coefficients

Parameters:

  • coeffs (pandas Series) – Importance values to plot.

  • top_n (int) – The number of features to display (Default=None). If None, display all. Otherwise, display top_n most important.

  • figsize (tuple) – figsize tuple for .plot.

  • legend_loc (str) – Location of the legend in the plot (Default=’best’).

  • threshold (float) – Threshold value for coloring the coefficients (Default=None).

  • color_lt (str) – Color for coefficients less than the threshold (Default=’darkred’).

  • color_gt (str) – Color for coefficients greater than the threshold (Default=’forestgreen’).

  • color_else (str) – Color for coefficients that do not meet the threshold (Default=’gray’).

  • label_thresh (str) – Label for the threshold line in the legend (Default=’Equally Likely’).

  • label_gt (str) – Label for coefficients greater than the threshold in the legend (Default=’More Likely’).

  • label_lt (str) – Label for coefficients less than the threshold in the legend (Default=’Less Likely’).

  • plot_kws (dict) – Additional keyword arguments accepted by pandas’ .plot method.

Returns:

Matplotlib axis object.

Return type:

matplotlib.axes._subplots.AxesSubplot

dojo_ds.insights.plot_importance(importances, top_n=None, figsize=(8, 6))[source]

Plots the importance of features in a horizontal bar chart.

Parameters: importances (pandas.Series): The importance values of the features. top_n (int, optional): The number of top most important features to plot. If None, all features will be plotted. Default is None. figsize (tuple, optional): The size of the figure. Default is (8, 6).

Returns: matplotlib.axes.Axes: The axes object of the plot.

dojo_ds.insights.plot_importance_color(importances, top_n=None, figsize=(8, 6), color_dict=None, ax=None)[source]

Plot the feature importances with optional color highlighting.

Parameters: - importances (pandas.Series): The feature importances. - top_n (int, optional): The number of top features to display. If None, all features will be displayed. Default is None. - figsize (tuple, optional): The figure size. Default is (8, 6). - color_dict (dict, optional): A dictionary mapping feature names to colors for highlighting. Default is None. - ax (matplotlib.axes.Axes, optional): The axes object to plot on. If None, a new figure and axes will be created. Default is None.

Returns: - ax (matplotlib.axes.Axes): The axes object containing the plot.

Example Use: fig, axes = plt.subplots(ncols=2, figsize=(20,8)) n = 20 # setting the # of features to use for both subplots

plot_importance_color(importances, top_n=n, ax=axes[0], color_dict= colors_top7) axes[0].set(title=’R.F. Importances’)

plot_importance_color(permutation_importances, top_n=n, ax=axes[1], color_dict=colors_top7) axes[1].set(title=’Permutation Importances’) fig.tight_layout()

dojo_ds.insights.plot_residuals(model, X_test_df, y_test, figsize=(12, 5))[source]

Plots a Q-Q Plot and residual plot for a statsmodels OLS regression.

Parameters: model (statsmodels.regression.linear_model.RegressionResultsWrapper): The fitted regression model. X_test_df (pandas.DataFrame): The test dataset features. y_test (array-like): The test dataset target variable. figsize (tuple, optional): The size of the figure. Defaults to (12,5).

Returns: None

dojo_ds.insights.summarize_df(df_)[source]

Summarizes a DataFrame by providing insights on column data types, null values, unique values, and numeric range.

Parameters: df_ (pandas.DataFrame): The DataFrame to be summarized.

Returns: pandas.DataFrame: A summary report DataFrame with the following columns:

  • ‘Column’: The column names of the DataFrame.

  • ‘dtype’: The data types of the columns.

  • ‘# null’: The number of null values in each column.

  • ‘null (%)’: The percentage of null values in each column.

  • ‘nunique’: The number of unique values in each column.

  • ‘min’: The minimum numeric value in each column.

  • ‘max’: The maximum numeric value in each column.

Example Usage: >> df = pd.read_csv(filename) >> summary = summarize_df(df)

dojo_ds.matplotlib_style module

dojo_ds.nlp module

nlp functions from Coding Dojo Online Data Science and Machine Learning 24-Week Program.

dojo_ds.nlp.batch_preprocess_texts(texts, nlp=None, remove_stopwords=True, remove_punct=True, use_lemmas=False, disable=['ner'], batch_size=50, n_process=-1)[source]

Efficiently preprocess a collection of texts using nlp.pipe()

Parameters:

  • texts (collection of strings) – Collection of texts to process (e.g. df[‘text’])

  • nlp (spacy pipe), optional) – Spacy nlp pipe. Defaults to None; if None, it creates a default ‘en_core_web_sm’ pipe.

  • remove_stopwords (bool, optional) – Controls stopword removal. Defaults to True.

  • remove_punct (bool, optional) – Controls punctuation removal. Defaults to True.

  • use_lemmas (bool, optional) – Lemmatize tokens. Defaults to False.

  • disable (list of strings, optional) – Named pipeline elements to disable. Defaults to [“ner”]: Used with nlp.pipe(disable=disable)

  • batch_size (int, optional) – Number of texts to process in a batch. Defaults to 50.

  • n_process (int, optional) – Number of CPU processors to use. Defaults to -1 (meaning all CPU cores).

Returns:

Processed texts as a list of tokens.

Return type:

list of tokens

dojo_ds.nlp.get_ngram_measures_finder(tokens, ngrams=2, measure='raw_freq', top_n=None, min_freq=1, words_colname='Words')[source]

Calculate n-gram measures for a given list of tokens.

Parameters: - tokens (list): List of tokens. - ngrams (int): Number of grams to consider (2 for bigrams, 3 for trigrams, 4 for quadgrams). Default is 2. - measure (str): Measure to calculate (‘raw_freq’ for raw frequency, ‘pmi’ for pointwise mutual information). Default is ‘raw_freq’. - top_n (int): Number of top n-grams to return. Default is None (return all n-grams). - min_freq (int): Minimum frequency threshold for n-grams. Default is 1. - words_colname (str): Column name for the n-grams in the resulting DataFrame. Default is ‘Words’.

Returns: - df_ngrams (DataFrame): DataFrame containing the n-grams and their corresponding measure values.

dojo_ds.nlp.make_custom_nlp(disable=['ner'], contractions=["don't", "can't", "couldn't", "you'd", "I'll"], stopwords_to_add=[], stopwords_to_remove=[], spacy_model='en_core_web_sm')[source]

Returns a custom spacy nlp pipeline.

Parameters:

  • disable (list, optional) – Names of pipe components to disable. Defaults to [“ner”].

  • contractions (list, optional) – List of contractions to add as special cases. Defaults to [“don’t”, “can’t”, “couldn’t”, “you’d”, “I’ll”].

  • stopwords_to_add (list, optional) – List of words to set as stopwords (word.is_stop=True)

  • stopwords_to_remove (list, optional) – List of words to remove from stopwords (word.is_stop=False)

  • spacy_model (str, optional) – Name or path of the spaCy model to load. Defaults to “en_core_web_sm”.

Returns:

spacy pipeline with special cases and updated nlp.Default.stopwords

Return type:

nlp pipeline

dojo_ds.nlp.make_text_vectorization_layer(train_ds, max_tokens=None, split='whitespace', standardize='lower_and_strip_punctuation', output_mode='int', output_sequence_length=None, ngrams=None, pad_to_max_tokens=False, verbose=True, **kwargs)[source]

Creates a text vectorization layer using TensorFlow’s TextVectorization class.

Parameters: - train_ds: The training dataset containing the text data. - max_tokens: The maximum number of tokens to keep in the vocabulary. - split: The method used to split the text into tokens. - standardize: The method used to standardize the text. - output_mode: The output mode of the layer. - output_sequence_length: The length of the output sequences. - ngrams: The n-grams to consider when tokenizing the text. - pad_to_max_tokens: Whether to pad the sequences to have the same length. - verbose: Whether to print the layer’s parameters. - **kwargs: Additional keyword arguments to pass to the TextVectorization class.

Returns: - text_vectorizer: The text vectorization layer. - int_to_str: A dictionary mapping integers to words in the vocabulary.

dojo_ds.nlp.preprocess_text(txt, nlp=None, remove_stopwords=True, remove_punct=True, use_lemmas=False)[source]

Preprocesses the given text by tokenizing and optionally removing stopwords, punctuation, and lemmatizing the tokens.

Parameters:

  • txt (str) – The text to be processed.

  • nlp (spacy pipe, optional) – The Spacy nlp pipe. Defaults to None. If None, it creates a default ‘en_core_web_sm’ pipe.

  • remove_stopwords (bool, optional) – Controls whether to remove stopwords. Defaults to True.

  • remove_punct (bool, optional) – Controls whether to remove punctuation. Defaults to True.

  • use_lemmas (bool, optional) – Controls whether to lemmatize tokens. Defaults to False.

Returns:

A list of tokens/lemmas after preprocessing.

Return type:

list

dojo_ds.nlp.reference_set_seed_keras(markdown=True)[source]

dojo_ds.time_series module

dojo_ds.time_series.get_adfuller_results(ts, alpha=0.05, label='adfuller', **kwargs)[source]
Uses statsmodels’ adfuller function to test a univariate time series for stationarity.
Null hypothesis:

The time series is NOT stationary. (It “has a unit root”.)

Interpretation:

a p-value less than alpha (.05) means the ts IS stationary. (We reject the null hypothesis that it is not stationary.)

Returns:

  • results (DataFrame) (DataFrame with the following columns/results:)

  • - “Test Statistic” (the adfuller test statistic.)

  • - “# of Lags Used” (The number of lags used in the calculation.)

  • - “# of Observations” (The number of observations used.)

  • - “p-value” (p-value for hypothesis test.)

  • - “alpha” (the significance level used for interpretin p-value)

  • - “sig/stationary?” (simplified interpretation of p-value)

  • ADFULLER DOCUMENTATION

  • For the full adfuller documentation, see

  • https (//www.statsmodels.org/dev/generated/statsmodels.tsa.stattools.adfuller.html)

dojo_ds.time_series.get_sig_lags(ts, type='ACF', nlags=None, alpha=0.5)[source]
dojo_ds.time_series.plot_acf_pacf(ts, nlags=40, figsize=(10, 5), annotate_sig=False, alpha=0.05, acf_kws={}, pacf_kws={}, annotate_seas=False, m=None, seas_color='black')[source]
dojo_ds.time_series.regression_metrics_ts(ts_true, ts_pred, label='', verbose=True, output_dict=False)[source]

dojo_ds.utils module

dojo_ds.utils.check_batch_size(dataset)[source]

Converts a file path into a clickable hyperlink.

Parameters: path (str): The file path to be converted. label (str, optional): The label to be displayed for the hyperlink. If not provided, the file path will be used as the label.

Returns: str: The clickable hyperlink.

Adapted from: https://www.geeksforgeeks.org/how-to-create-a-table-with-clickable-hyperlink-to-a-local-file-in-pandas/

dojo_ds.utils.column_report(df, index_col=None, sort_column='iloc', ascending=True, interactive=False, return_df=False)[source]

Displays a DataFrame summary of each column’s: - name, iloc, dtypes, null value count & %, # of 0’s, min, max, med, mean, etc

Parameters:

  • df (DataFrame) – The DataFrame to report on.

  • index_col (str, optional) – The column to set as the index. Defaults to None.

  • sort_column (str, optional) – The column to sort the report by. Defaults to ‘iloc’.

  • ascending (bool, optional) – Whether to sort the report in ascending order. Defaults to True.

  • interactive (bool, optional) – Whether to enable interactive sorting. Defaults to False.

  • return_df (bool, optional) – Whether to return the non-styled version of the report DataFrame. Defaults to False.

Returns:

The non-styled version of the displayed report DataFrame.

Return type:

column_report (DataFrame)

dojo_ds.utils.create_directories_from_paths(nested_dict)[source]

OpenAI. (2023). ChatGPT [Large language model]. https://chat.openai.com Recursively create directories for file paths in a nested dictionary.

Parameters: nested_dict (dict): The nested dictionary containing file paths.

dojo_ds.utils.deep_getsizeof(obj, seen=None, unit='MB', top_level=True, return_size=True)[source]

# Function provided by OpenAI’s ChatGPT # Date: November 1, 2023

Calculate the deep size of a Python object including nested objects.

Parameters:

  • obj (object) – The Python object whose size is to be calculated.

  • seen (set, optional) – A set of object ids to handle circular references. Defaults to None.

  • unit (str, optional) – The unit in which to return the size. Options are ‘B’ for Bytes, ‘KB’ for Kilobytes, ‘MB’ for Megabytes, ‘GB’ for Gigabytes. Defaults to ‘B’.

  • top_level (bool, optional) – Whether the function is called at the top-level (not recursively). Defaults to True.

Returns:

The size of the object in the unit specified.

Return type:

float

Example

>>> my_dict = {'key1': 'value1', 'key2': [1, 2, 3], 'key3': {'inner_key': 'value'}}
>>> deep_getsizeof(my_dict, unit='KB')
dojo_ds.utils.get_attributes(obj, private=False)[source]

Retrieves a list of all non-private attributes (default) from inside of obj. - If private==False: only returns methods whose names do NOT start with a ‘_’

Parameters:

  • obj (object) – Object to retrieve attributes from.

  • private (bool, optional) – Whether to retrieve private attributes or public. Defaults to False.

Returns:

The names of all the retrieved attributes.

Return type:

list

dojo_ds.utils.get_filesize(fpath, unit='MB')[source]
dojo_ds.utils.get_methods(obj, private=False)[source]

Retrieves a list of all non-private methods (default) from inside of obj.

Parameters:

  • obj (object) – Object to retrieve methods from.

  • private (bool, optional) – Whether to retrieve private methods or public. Defaults to False, which retrieves only public methods.

Returns:

The names of all the retrieved methods.

Return type:

list

Examples

>>> class MyClass:
...     def public_method(self):
...         pass
...     def _private_method(self):
...         pass
...
>>> obj = MyClass()
>>> get_methods(obj)
['public_method']
>>> get_methods(obj, private=True)
['public_method', '_private_method']
dojo_ds.utils.get_or_print_filesize(fpath, unit='MB', print_or_return='print')[source]

Get the file size as a string, converted to the requested unit(B,KB, MB, GB)

Parameters:

  • fpath (string) – file to analyze

  • unit (str, optional) – unit for conversion. Defaults to “MB”.

  • print_or_return (str, optional) – Controls if string is returned or printed. Defaults to ‘print’.

Returns:

file size + units

Return type:

string

Raises:

FileNotFoundError – If the specified file does not exist.

dojo_ds.utils.inspect_file(fname, units='mb', verbose=False)[source]

Returns a dictionary with detailed file information including: - File name, extension, file size, date created, date modified, etc. :param fname: filepath :type fname: str :param units: Units for fileszize. (Options are “kb’,’mb’,’gb’). Defaults to ‘mb’. :type units: str, optional

Returns:

dictionary with info

Return type:

dict

dojo_ds.utils.inspect_variables(local_vars=None, sort_col='size', exclude_funcs_mods=True, top_n=10, return_df=False, always_display=True, show_how_to_delete=False, print_names=False)[source]

Displays a dataframe of all variables and their size in memory, with the largest variables at the top.

Parameters:

  • local_vars (locals() – Must call locals() as first argument.

  • sort_col (str, optional) – column to sort by. Defaults to ‘size’.

  • top_n (int, optional) – how many vars to show. Defaults to 10.

  • return_df (bool, optional) – If True, return df instead of just showing df.Defaults to False.

  • always_display (bool, optional) – Display df even if returned. Defaults to True.

  • show_how_to_delete (bool, optional) – Prints out code to copy-paste into cell to del vars. Defaults to False.

  • print_names (bool, optional) – [description]. Defaults to False.

Raises:

Exception – if locals() not passed as first arg

Example Usage: # Must pass in local variables >> inspect_variables(locals()) # To see command to delete list of vars” >> inspect_variables(locals(),show_how_to_delete=True)

dojo_ds.utils.preview_ds(train_ds, n_rows=3, n_tokens=500)[source]
dojo_ds.utils.print_and_convert_size(size, unit='B')[source]

# Function provided by OpenAI’s ChatGPT # Date: November 1, 2023 Convert and print the size into the specified unit.

Parameters:

  • size (float) – The size in bytes.

  • unit (str, optional) – The unit in which to print and return the size. Options are ‘B’ for Bytes, ‘KB’ for Kilobytes, ‘MB’ for Megabytes, ‘GB’ for Gigabytes. Defaults to ‘B’.

Returns:

The size in the unit specified.

Return type:

float

dojo_ds.utils.read_and_fix_json(JSON_FILE)[source]

Attempts to read in json file of records and fixes the final character to end with a ] if it errors.

Parameters:

JSON_FILE (str) – filepath of JSON file

Returns:

the corrected data from the bad json file

Return type:

DataFrame

dojo_ds.utils.reference_set_seed_keras(markdown=True)[source]
dojo_ds.utils.show_del_me_code(called_by_inspect_vars=False)[source]

Prints code to copy and paste into a cell to delete vars using a list of their names. Companion function inspect_variables(locals(),print_names=True) will provide var names tocopy/paste

dojo_ds.utils.write_json(new_data, filename)[source]

Adapted from: https://www.geeksforgeeks.org/append-to-json-file-using-python/