Hyperparameter and structural time series component selection optimisation to ensure model convergence
Budget: $250 – $750 USD
I need a freelancer to optimize the hyperparameters and structural time series component selection of my model to ensure convergence whilst maintaining model accuracy. The project requires expertise in Python programming language. I have a specific dataset in mind that will be used for the project. Ideal skills and experience for this job include:
- Strong background in machine learning and data analysis
- Proficiency in Python programming language
- Experience with hyperparameter optimization and time series analysis
- Familiarity with structural time series models
- Ability to work with large datasets and optimize computational time
Requirements Specification:
Objective:
Develop a time series forecasting model using TensorFlow Probability Structural Time Series (TFP STS) to predict future values of a given dataset.
Requirements:
1/. Loss Function Monitoring
- Implement a mechanism to track the training loss during model optimization.
- Ensure that the loss function decreases over time and stabilizes, indicating convergence.
- Include periodic logging and visualization of the training loss.
2/. Trace Plots:
- Generate trace plots for all model parameters during Markov Chain Monte Carlo (MCMC) sampling.
- Verify that the traces are stationary and do not exhibit obvious trends, indicative of convergence.
- Include functionality to visualize and save trace plots for further analysis.
3/. Autocorrelation Plots:
- Create autocorrelation plots for MCMC samples of each model parameter.
- Confirm that autocorrelation is low, suggesting that the chain has mixed well and converged.
- Implement a feature to visualize and save autocorrelation plots for review.
4/. Effective Sample Size (ESS):
- Calculate the effective sample size for each model parameter.
- Ensure that ESS values are sufficiently large, indicating better convergence.
- Provide a summary of ESS values for all parameters and include functionality to export these values.
5/. R-hat (Gelman-Rubin Diagnostic):
- Compute the R-hat statistic for each model parameter.
- Verify that R-hat values are close to 1, signaling good convergence.
- Implement a summary report of R-hat values for all parameters and include an export feature.
6/. Posterior Predictive Checks:
- Perform posterior predictive checks by generating simulated data from the fitted model.
- Compare simulated data to the observed data in terms of summary statistics and overall structure.
- Ensure that the model demonstrates satisfactory convergence by producing realistic simulations.
- Include functionality to visualize and save posterior predictive check results.
7/. Widely Applicable Information Criterion (WAIC) and Deviance Information Criterion (DIC):
- Calculate the WAIC and DIC for the fitted Bayesian model.
- Ensure that the WAIC and DIC values are reasonably low, indicating a good balance between model fit and complexity.
- Track the WAIC and DIC values during model selection, comparing different model configurations to choose the best one.
- Provide a summary report of WAIC and DIC values for different models and include functionality to export these values.
Deliverables:
- A well-documented Jupyter Notebook or Python script demonstrating the entire model development process.
- A summary report that includes the convergence diagnostic results (trace plots, autocorrelation plots, ESS, R-hat values, and posterior predictive checks).
- A serialized version of the trained model for future use.
- A user guide detailing the usage of the provided code and explanation of the convergence diagnostics.
Note:
The notebook already contains many of these metrics including MAPE (posterior predictive checks), R-hat and a traceplot. Please ensure these metrics are shown for each parameter. In other words, please ensure that the R-hat, traceplot, ESS, WAIC and DIC shows values for each parameter.
- Strong background in machine learning and data analysis
- Proficiency in Python programming language
- Experience with hyperparameter optimization and time series analysis
- Familiarity with structural time series models
- Ability to work with large datasets and optimize computational time
Requirements Specification:
Objective:
Develop a time series forecasting model using TensorFlow Probability Structural Time Series (TFP STS) to predict future values of a given dataset.
Requirements:
1/. Loss Function Monitoring
- Implement a mechanism to track the training loss during model optimization.
- Ensure that the loss function decreases over time and stabilizes, indicating convergence.
- Include periodic logging and visualization of the training loss.
2/. Trace Plots:
- Generate trace plots for all model parameters during Markov Chain Monte Carlo (MCMC) sampling.
- Verify that the traces are stationary and do not exhibit obvious trends, indicative of convergence.
- Include functionality to visualize and save trace plots for further analysis.
3/. Autocorrelation Plots:
- Create autocorrelation plots for MCMC samples of each model parameter.
- Confirm that autocorrelation is low, suggesting that the chain has mixed well and converged.
- Implement a feature to visualize and save autocorrelation plots for review.
4/. Effective Sample Size (ESS):
- Calculate the effective sample size for each model parameter.
- Ensure that ESS values are sufficiently large, indicating better convergence.
- Provide a summary of ESS values for all parameters and include functionality to export these values.
5/. R-hat (Gelman-Rubin Diagnostic):
- Compute the R-hat statistic for each model parameter.
- Verify that R-hat values are close to 1, signaling good convergence.
- Implement a summary report of R-hat values for all parameters and include an export feature.
6/. Posterior Predictive Checks:
- Perform posterior predictive checks by generating simulated data from the fitted model.
- Compare simulated data to the observed data in terms of summary statistics and overall structure.
- Ensure that the model demonstrates satisfactory convergence by producing realistic simulations.
- Include functionality to visualize and save posterior predictive check results.
7/. Widely Applicable Information Criterion (WAIC) and Deviance Information Criterion (DIC):
- Calculate the WAIC and DIC for the fitted Bayesian model.
- Ensure that the WAIC and DIC values are reasonably low, indicating a good balance between model fit and complexity.
- Track the WAIC and DIC values during model selection, comparing different model configurations to choose the best one.
- Provide a summary report of WAIC and DIC values for different models and include functionality to export these values.
Deliverables:
- A well-documented Jupyter Notebook or Python script demonstrating the entire model development process.
- A summary report that includes the convergence diagnostic results (trace plots, autocorrelation plots, ESS, R-hat values, and posterior predictive checks).
- A serialized version of the trained model for future use.
- A user guide detailing the usage of the provided code and explanation of the convergence diagnostics.
Note:
The notebook already contains many of these metrics including MAPE (posterior predictive checks), R-hat and a traceplot. Please ensure these metrics are shown for each parameter. In other words, please ensure that the R-hat, traceplot, ESS, WAIC and DIC shows values for each parameter.