A. Correct.
( SHOW ANSWER ) To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use the applyInPandas function. This function allows you to apply a Python function on each group of a DataFrame and return a DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance. prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) ) In this code: groupby("device_id"): Groups the DataFrame by the "device_id" column. applyInPandas(apply_model, schema=apply_return_schema): Applies the apply_model function to each group and specifies the schema of the return DataFrame. Reference: PySpark Pandas UDFs Documentation

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use the applyInPandas function. This function allows you to apply a Python function on each group of a DataFrame and return a DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance. prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) ) In this code: groupby("device_id"): Groups the DataFrame by the "device_id" column. applyInPandas(apply_model, schema=apply_return_schema): Applies the apply_model function to each group and specifies the schema of the return DataFrame. Reference: PySpark Pandas UDFs Documentation

C. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use the applyInPandas function. This function allows you to apply a Python function on each group of a DataFrame and return a DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance. prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) ) In this code: groupby("device_id"): Groups the DataFrame by the "device_id" column. applyInPandas(apply_model, schema=apply_return_schema): Applies the apply_model function to each group and specifies the schema of the return DataFrame. Reference: PySpark Pandas UDFs Documentation

D. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use the applyInPandas function. This function allows you to apply a Python function on each group of a DataFrame and return a DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance. prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) ) In this code: groupby("device_id"): Groups the DataFrame by the "device_id" column. applyInPandas(apply_model, schema=apply_return_schema): Applies the apply_model function to each group and specifies the schema of the return DataFrame. Reference: PySpark Pandas UDFs Documentation

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When the goal is to maximize the identification of positive cases in a classification task, the metric of interest is Recall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e., the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of performance measurement and are not specifically focused on maximizing the detection of positive cases alone. Reference: Classification Metrics in Machine Learning (Understanding Recall).

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When the goal is to maximize the identification of positive cases in a classification task, the metric of interest is Recall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e., the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of performance measurement and are not specifically focused on maximizing the detection of positive cases alone. Reference: Classification Metrics in Machine Learning (Understanding Recall).

C. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When the goal is to maximize the identification of positive cases in a classification task, the metric of interest is Recall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e., the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of performance measurement and are not specifically focused on maximizing the detection of positive cases alone. Reference: Classification Metrics in Machine Learning (Understanding Recall).

D. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When the goal is to maximize the identification of positive cases in a classification task, the metric of interest is Recall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e., the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of performance measurement and are not specifically focused on maximizing the detection of positive cases alone. Reference: Classification Metrics in Machine Learning (Understanding Recall).

E. Correct.
( SHOW ANSWER ) When the goal is to maximize the identification of positive cases in a classification task, the metric of interest is Recall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e., the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of performance measurement and are not specifically focused on maximizing the detection of positive cases alone. Reference: Classification Metrics in Machine Learning (Understanding Recall).

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Ensemble learning is a machine learning technique that involves combining several models to solve a particular problem. The scenario described fits the concept of ensemble learning, where two models, each performing well under different conditions, are combined to create a more robust model. This approach often leads to better performance as it combines the strengths of multiple models. Reference Introduction to Ensemble Learning: https://machinelearningmastery.com/ensemble-machine- learning-algorithms-python-scikit-learn/

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Ensemble learning is a machine learning technique that involves combining several models to solve a particular problem. The scenario described fits the concept of ensemble learning, where two models, each performing well under different conditions, are combined to create a more robust model. This approach often leads to better performance as it combines the strengths of multiple models. Reference Introduction to Ensemble Learning: https://machinelearningmastery.com/ensemble-machine- learning-algorithms-python-scikit-learn/

C. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Ensemble learning is a machine learning technique that involves combining several models to solve a particular problem. The scenario described fits the concept of ensemble learning, where two models, each performing well under different conditions, are combined to create a more robust model. This approach often leads to better performance as it combines the strengths of multiple models. Reference Introduction to Ensemble Learning: https://machinelearningmastery.com/ensemble-machine- learning-algorithms-python-scikit-learn/

D. Correct.
( SHOW ANSWER ) Ensemble learning is a machine learning technique that involves combining several models to solve a particular problem. The scenario described fits the concept of ensemble learning, where two models, each performing well under different conditions, are combined to create a more robust model. This approach often leads to better performance as it combines the strengths of multiple models. Reference Introduction to Ensemble Learning: https://machinelearningmastery.com/ensemble-machine- learning-algorithms-python-scikit-learn/

E. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Ensemble learning is a machine learning technique that involves combining several models to solve a particular problem. The scenario described fits the concept of ensemble learning, where two models, each performing well under different conditions, are combined to create a more robust model. This approach often leads to better performance as it combines the strengths of multiple models. Reference Introduction to Ensemble Learning: https://machinelearningmastery.com/ensemble-machine- learning-algorithms-python-scikit-learn/

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes: Hyperparameter 1: [2, 5, 10] (3 values) Hyperparameter 2: [50, 100] (2 values) The total number of combinations is the product of the number of values for each hyperparameter: 3 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=63 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=6 With 3-fold cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will be: 6 (combinations)×3 (folds)=186 (combinations)×3 (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation. Therefore, 6 models can be trained in parallel. Reference: Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes: Hyperparameter 1: [2, 5, 10] (3 values) Hyperparameter 2: [50, 100] (2 values) The total number of combinations is the product of the number of values for each hyperparameter: 3 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=63 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=6 With 3-fold cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will be: 6 (combinations)×3 (folds)=186 (combinations)×3 (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation. Therefore, 6 models can be trained in parallel. Reference: Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

C. Correct.
( SHOW ANSWER ) To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes: Hyperparameter 1: [2, 5, 10] (3 values) Hyperparameter 2: [50, 100] (2 values) The total number of combinations is the product of the number of values for each hyperparameter: 3 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=63 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=6 With 3-fold cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will be: 6 (combinations)×3 (folds)=186 (combinations)×3 (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation. Therefore, 6 models can be trained in parallel. Reference: Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

D. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes: Hyperparameter 1: [2, 5, 10] (3 values) Hyperparameter 2: [50, 100] (2 values) The total number of combinations is the product of the number of values for each hyperparameter: 3 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=63 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=6 With 3-fold cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will be: 6 (combinations)×3 (folds)=186 (combinations)×3 (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation. Therefore, 6 models can be trained in parallel. Reference: Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When using the Hyperopt library with fmin, the goal is to find the minimum of the objective function. Since you are using cross_val_score to calculate the R2 score which is a measure of the proportion of the variance for a dependent variable that's explained by an independent variable(s) in a regression model, higher values are better. However, fmin seeks to minimize the objective function, so to align with fmin's goal, you should return the negative of the R2 score (-r2). This way, by minimizing the negative R2, fmin is effectively maximizing the R2 score, which can lead to a more accurate model. Reference Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit- learn.org/stable/modules/model_evaluation.html

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When using the Hyperopt library with fmin, the goal is to find the minimum of the objective function. Since you are using cross_val_score to calculate the R2 score which is a measure of the proportion of the variance for a dependent variable that's explained by an independent variable(s) in a regression model, higher values are better. However, fmin seeks to minimize the objective function, so to align with fmin's goal, you should return the negative of the R2 score (-r2). This way, by minimizing the negative R2, fmin is effectively maximizing the R2 score, which can lead to a more accurate model. Reference Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit- learn.org/stable/modules/model_evaluation.html

C. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When using the Hyperopt library with fmin, the goal is to find the minimum of the objective function. Since you are using cross_val_score to calculate the R2 score which is a measure of the proportion of the variance for a dependent variable that's explained by an independent variable(s) in a regression model, higher values are better. However, fmin seeks to minimize the objective function, so to align with fmin's goal, you should return the negative of the R2 score (-r2). This way, by minimizing the negative R2, fmin is effectively maximizing the R2 score, which can lead to a more accurate model. Reference Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit- learn.org/stable/modules/model_evaluation.html

D. Correct.
( SHOW ANSWER ) When using the Hyperopt library with fmin, the goal is to find the minimum of the objective function. Since you are using cross_val_score to calculate the R2 score which is a measure of the proportion of the variance for a dependent variable that's explained by an independent variable(s) in a regression model, higher values are better. However, fmin seeks to minimize the objective function, so to align with fmin's goal, you should return the negative of the R2 score (-r2). This way, by minimizing the negative R2, fmin is effectively maximizing the R2 score, which can lead to a more accurate model. Reference Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit- learn.org/stable/modules/model_evaluation.html

E. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) When using the Hyperopt library with fmin, the goal is to find the minimum of the objective function. Since you are using cross_val_score to calculate the R2 score which is a measure of the proportion of the variance for a dependent variable that's explained by an independent variable(s) in a regression model, higher values are better. However, fmin seeks to minimize the objective function, so to align with fmin's goal, you should return the negative of the R2 score (-r2). This way, by minimizing the negative R2, fmin is effectively maximizing the R2 score, which can lead to a more accurate model. Reference Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit- learn.org/stable/modules/model_evaluation.html

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) By creating a binary feature variable for each feature with missing values to indicate whether a value has been imputed, the data scientist can preserve information about the original state of the data. This approach maintains the integrity of the dataset by marking which values are original and which are synthetic (imputed). Here are the steps to implement this approach: Identify Missing Values: Determine which features contain missing values. Impute Missing Values: Continue with median imputation or choose another method (mean, mode, regression, etc.) to fill missing values. Create Indicator Variables: For each feature that had missing values, add a new binary feature. This feature should be '1' if the original value was missing and imputed, and '0' otherwise. Data Integration: Integrate these new binary features into the existing dataset. This maintains a record of where data imputation occurred, allowing models to potentially weight these observations differently. Model Adjustment: Adjust machine learning models to account for these new features, which might involve considering interactions between these binary indicators and other features. Reference "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari (O'Reilly Media, 2018), especially the sections on handling missing data. Scikit-learn documentation on imputing missing values: https://scikit- learn.org/stable/modules/impute.html

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) By creating a binary feature variable for each feature with missing values to indicate whether a value has been imputed, the data scientist can preserve information about the original state of the data. This approach maintains the integrity of the dataset by marking which values are original and which are synthetic (imputed). Here are the steps to implement this approach: Identify Missing Values: Determine which features contain missing values. Impute Missing Values: Continue with median imputation or choose another method (mean, mode, regression, etc.) to fill missing values. Create Indicator Variables: For each feature that had missing values, add a new binary feature. This feature should be '1' if the original value was missing and imputed, and '0' otherwise. Data Integration: Integrate these new binary features into the existing dataset. This maintains a record of where data imputation occurred, allowing models to potentially weight these observations differently. Model Adjustment: Adjust machine learning models to account for these new features, which might involve considering interactions between these binary indicators and other features. Reference "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari (O'Reilly Media, 2018), especially the sections on handling missing data. Scikit-learn documentation on imputing missing values: https://scikit- learn.org/stable/modules/impute.html

C. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) By creating a binary feature variable for each feature with missing values to indicate whether a value has been imputed, the data scientist can preserve information about the original state of the data. This approach maintains the integrity of the dataset by marking which values are original and which are synthetic (imputed). Here are the steps to implement this approach: Identify Missing Values: Determine which features contain missing values. Impute Missing Values: Continue with median imputation or choose another method (mean, mode, regression, etc.) to fill missing values. Create Indicator Variables: For each feature that had missing values, add a new binary feature. This feature should be '1' if the original value was missing and imputed, and '0' otherwise. Data Integration: Integrate these new binary features into the existing dataset. This maintains a record of where data imputation occurred, allowing models to potentially weight these observations differently. Model Adjustment: Adjust machine learning models to account for these new features, which might involve considering interactions between these binary indicators and other features. Reference "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari (O'Reilly Media, 2018), especially the sections on handling missing data. Scikit-learn documentation on imputing missing values: https://scikit- learn.org/stable/modules/impute.html

D. Correct.
( SHOW ANSWER ) By creating a binary feature variable for each feature with missing values to indicate whether a value has been imputed, the data scientist can preserve information about the original state of the data. This approach maintains the integrity of the dataset by marking which values are original and which are synthetic (imputed). Here are the steps to implement this approach: Identify Missing Values: Determine which features contain missing values. Impute Missing Values: Continue with median imputation or choose another method (mean, mode, regression, etc.) to fill missing values. Create Indicator Variables: For each feature that had missing values, add a new binary feature. This feature should be '1' if the original value was missing and imputed, and '0' otherwise. Data Integration: Integrate these new binary features into the existing dataset. This maintains a record of where data imputation occurred, allowing models to potentially weight these observations differently. Model Adjustment: Adjust machine learning models to account for these new features, which might involve considering interactions between these binary indicators and other features. Reference "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari (O'Reilly Media, 2018), especially the sections on handling missing data. Scikit-learn documentation on imputing missing values: https://scikit- learn.org/stable/modules/impute.html

E. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) By creating a binary feature variable for each feature with missing values to indicate whether a value has been imputed, the data scientist can preserve information about the original state of the data. This approach maintains the integrity of the dataset by marking which values are original and which are synthetic (imputed). Here are the steps to implement this approach: Identify Missing Values: Determine which features contain missing values. Impute Missing Values: Continue with median imputation or choose another method (mean, mode, regression, etc.) to fill missing values. Create Indicator Variables: For each feature that had missing values, add a new binary feature. This feature should be '1' if the original value was missing and imputed, and '0' otherwise. Data Integration: Integrate these new binary features into the existing dataset. This maintains a record of where data imputation occurred, allowing models to potentially weight these observations differently. Model Adjustment: Adjust machine learning models to account for these new features, which might involve considering interactions between these binary indicators and other features. Reference "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari (O'Reilly Media, 2018), especially the sections on handling missing data. Scikit-learn documentation on imputing missing values: https://scikit- learn.org/stable/modules/impute.html

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging involves training multiple models independently on different random subsets of the data and then combining their predictions. Random Forests consist of many decision trees trained on random subsets of the training data and features, and their predictions are averaged to improve accuracy and control overfitting. This method enhances model robustness and predictive performance. Reference: Ensemble Methods in Machine Learning (Understanding Bagging and Random Forests).

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging involves training multiple models independently on different random subsets of the data and then combining their predictions. Random Forests consist of many decision trees trained on random subsets of the training data and features, and their predictions are averaged to improve accuracy and control overfitting. This method enhances model robustness and predictive performance. Reference: Ensemble Methods in Machine Learning (Understanding Bagging and Random Forests).

C. Correct.
( SHOW ANSWER ) Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging involves training multiple models independently on different random subsets of the data and then combining their predictions. Random Forests consist of many decision trees trained on random subsets of the training data and features, and their predictions are averaged to improve accuracy and control overfitting. This method enhances model robustness and predictive performance. Reference: Ensemble Methods in Machine Learning (Understanding Bagging and Random Forests).

D. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging involves training multiple models independently on different random subsets of the data and then combining their predictions. Random Forests consist of many decision trees trained on random subsets of the training data and features, and their predictions are averaged to improve accuracy and control overfitting. This method enhances model robustness and predictive performance. Reference: Ensemble Methods in Machine Learning (Understanding Bagging and Random Forests).

E. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging involves training multiple models independently on different random subsets of the data and then combining their predictions. Random Forests consist of many decision trees trained on random subsets of the training data and features, and their predictions are averaged to improve accuracy and control overfitting. This method enhances model robustness and predictive performance. Reference: Ensemble Methods in Machine Learning (Understanding Bagging and Random Forests).

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Using an iterator in the pandas_udf ensures that the model only needs to be loaded once per executor rather than once per batch. This approach reduces the overhead associated with repeatedly loading the model during the inference process, leading to more efficient and faster predictions. The data will be distributed across multiple executors, but each executor will load the model only once, optimizing the inference process. Reference: Databricks documentation on pandas UDFs: Pandas UDFs

B. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Using an iterator in the pandas_udf ensures that the model only needs to be loaded once per executor rather than once per batch. This approach reduces the overhead associated with repeatedly loading the model during the inference process, leading to more efficient and faster predictions. The data will be distributed across multiple executors, but each executor will load the model only once, optimizing the inference process. Reference: Databricks documentation on pandas UDFs: Pandas UDFs

C. Correct.
( SHOW ANSWER ) Using an iterator in the pandas_udf ensures that the model only needs to be loaded once per executor rather than once per batch. This approach reduces the overhead associated with repeatedly loading the model during the inference process, leading to more efficient and faster predictions. The data will be distributed across multiple executors, but each executor will load the model only once, optimizing the inference process. Reference: Databricks documentation on pandas UDFs: Pandas UDFs

D. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) Using an iterator in the pandas_udf ensures that the model only needs to be loaded once per executor rather than once per batch. This approach reduces the overhead associated with repeatedly loading the model during the inference process, leading to more efficient and faster predictions. The data will be distributed across multiple executors, but each executor will load the model only once, optimizing the inference process. Reference: Databricks documentation on pandas UDFs: Pandas UDFs

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration or other changes in the environment. Correct approach: Split the data. Write the split data to persistent storage (e.g., HDFS, S3). Load the data from storage for each model training session. train_df, test_df = spark_df.randomSplit([0.8, 0.2], seed=42) train_df.write.parquet("path/to/train_df.parquet") test_df.write.parquet("path/to/test_df.parquet") # Later, load the data train_df = spark.read.parquet("path/to/train_df.parquet") test_df = spark.read.parquet("path/to/test_df.parquet") Reference: Spark DataFrameWriter Documentation

B. Correct.
( SHOW ANSWER ) To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration or other changes in the environment. Correct approach: Split the data. Write the split data to persistent storage (e.g., HDFS, S3). Load the data from storage for each model training session. train_df, test_df = spark_df.randomSplit([0.8, 0.2], seed=42) train_df.write.parquet("path/to/train_df.parquet") test_df.write.parquet("path/to/test_df.parquet") # Later, load the data train_df = spark.read.parquet("path/to/train_df.parquet") test_df = spark.read.parquet("path/to/test_df.parquet") Reference: Spark DataFrameWriter Documentation

C. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration or other changes in the environment. Correct approach: Split the data. Write the split data to persistent storage (e.g., HDFS, S3). Load the data from storage for each model training session. train_df, test_df = spark_df.randomSplit([0.8, 0.2], seed=42) train_df.write.parquet("path/to/train_df.parquet") test_df.write.parquet("path/to/test_df.parquet") # Later, load the data train_df = spark.read.parquet("path/to/train_df.parquet") test_df = spark.read.parquet("path/to/test_df.parquet") Reference: Spark DataFrameWriter Documentation

D. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration or other changes in the environment. Correct approach: Split the data. Write the split data to persistent storage (e.g., HDFS, S3). Load the data from storage for each model training session. train_df, test_df = spark_df.randomSplit([0.8, 0.2], seed=42) train_df.write.parquet("path/to/train_df.parquet") test_df.write.parquet("path/to/test_df.parquet") # Later, load the data train_df = spark.read.parquet("path/to/train_df.parquet") test_df = spark.read.parquet("path/to/test_df.parquet") Reference: Spark DataFrameWriter Documentation

A. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To apply the Pandas UDF predict to each record of a Spark DataFrame, you use the mapInPandas method. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predict in this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments or incorrect function calls. Reference: PySpark DataFrame documentation (Using mapInPandas with UDFs).

B. Correct.
( SHOW ANSWER ) To apply the Pandas UDF predict to each record of a Spark DataFrame, you use the mapInPandas method. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predict in this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments or incorrect function calls. Reference: PySpark DataFrame documentation (Using mapInPandas with UDFs).

C. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To apply the Pandas UDF predict to each record of a Spark DataFrame, you use the mapInPandas method. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predict in this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments or incorrect function calls. Reference: PySpark DataFrame documentation (Using mapInPandas with UDFs).

D. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To apply the Pandas UDF predict to each record of a Spark DataFrame, you use the mapInPandas method. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predict in this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments or incorrect function calls. Reference: PySpark DataFrame documentation (Using mapInPandas with UDFs).

E. Incorrect.
This option does not match the PDF answer key. ( SHOW ANSWER ) To apply the Pandas UDF predict to each record of a Spark DataFrame, you use the mapInPandas method. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predict in this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments or incorrect function calls. Reference: PySpark DataFrame documentation (Using mapInPandas with UDFs).