install.packages(rjwsacruncher)Cruncher and quality report
In this chapter
The sections below describe how to
automate the Seasonal adjustment estimation process
update a workspace when new data is available
export output (series, diagnostics, parameters)
generate a quality report usable for selective editing (manual fine tuning)
Automate estimation with the cruncher
The cruncher is an additional “executable” module. It can be launched via R or SAS for example.
Objectives of the cruncher:
update a JDemetra+ workspace (with a selected revision policy)
export the results (series and diagnostics),
without having to open the graphical interface and operate manually. Suitable for a production process.
Installation procedure
- Download the cruncher
Available here https://github.com/jdemetra/jwsacruncher/releases
Click on the zip code line of the latest release
- Unzip locally (or on server)
Help pages
Documentation is available here or click on the wiki icon on the Github page https://github.com/jdemetra/jwsacruncher/wiki
Running the cruncher in R
Two R packages are currently available
rjwsacruncher on CRAN: workspace update and output production
Cruncher (https://github.com/InseeFr/JDCruncheR): same functions as rjwsacruncher but adds a quality report
Installation
- rjwsacruncher
- JDCruncheR package: download the .zip or .tar.gz file from https://github.com/InseeFr/JDCruncheR/releases.
Additional packages needed
install.packages(c("XLConnect", "XML"))Loading
library("JDCruncheR")or
library("rjwsacruncher")Connecting the cruncher module
To connect the cruncher to the R package, the path to the bin directory containing the cruncher.bat file must be specified. This directory is available once the zip file has been unzipped.
options(
cruncher_bin_directory =
"C:/Software/jwsacruncher-2.2.3/jdemetra-cli-2.2.3/bin"
)- checking the current value
getOption("cruncher_bin_directory")Updating a workspace
The functions described in this section are identical for both packages.
Running estimations
The general context: two use cases
- Run first estimation of seasonally adjusted series (from raw series and parameters contained in the workspace)
cruncher_and_param(
workspace = "D:/my_folder/my_ws.xml",
rename_multi_documents = FALSE,
policy = "complete", # name of the revision policy
log = my_log_file.txt
)- Apply a revision policy to updated raw series
The function cruncher_and_param() allows to do that
cruncher_and_param(
workspace = "D:/my_folder/my_ws.xml",
rename_multi_documents = FALSE,
policy = "lastoutliers", # name of the revision policy
log = my_log_file.txt
)To use the documentation, compute help() or ?function:
?cruncher_and_param
help(cruncher_and_param)Before running SA estimations, set the export options.
Configuring output options
After updating the workspace with the selected revision policies, the cruncher generates output - series (csv files) - diagnostics and parameters (demetra_m.csv file)
These files will be created in the workspace’s repository, sub-repository ’Output”
path <- "My_Workspace/Output/SAProcessing"Selecting time series to export
# returns names of the currently exported series
getOption("default_tsmatrix_series")
# example of setting this option
options(default_tsmatrix_series = c("sa", "sa_f"))
# only seasonally adjsuted series ("sa") and its forecasts ("sa_f") will be exportedSelecting diagnostics and parameters to exoprt
# returns names of the currently exported diagnostics and parameters
getOption("default_matrix_item")
# example of setting this option
options(default_matrix_item = c(
"likelihood.aic",
"likelihood.aicc",
"likelihood.bic",
"likelihood.bicc"
))Quality report with JDCruncheR
The JDCruncheR package also:
computes a quality score
creates a quality report from the diagnostics produced by JDemetra+
Main steps
The three main functions of the package are:
extract_QRto extract the quality report from the csv file (demetra_m.csv) that contains all JD+ diagnostics;compute_scoreto compute a weighted score based on the diagnosticsexport_xlsxto export the quality report.
# choose the demetra_m.csv file generated by the cruncher
QR <- extract_QR()
QR
?compute_score # to see how the score is calculated (formula)
QR <- compute_score(QR,
n_contrib_score = 3
)
QR
QR <- sort(QR, decreasing = TRUE, sort_variables = "score")
export_xlsx(QR,
file_name = "U:/quality_report.xls"
)Piling up results
When working with several workspaces or Seasonal adjustment processings (SAP) within a given workspace, quality reports can be piled up with the function rbind() or by creating a mQR_matrix object with the function mQR_matrix()
QR1 <- extract_QR()
QR2 <- extract_QR()
mQR <- mQR_matrix(QR1, QR2)
mQR
# naming each object
names(mQR) <- c("report_1", "report_2")
# Equivalent to:
mQR <- mQR_matrix(report_1 = QR1, report_2 = QR2)
mQR
# score calculation for all reports
mQR <- compute_score(mQR,
n_contrib_score = 3
)
export_xlsx(mQR,
export_dir = "U:/"
)Conditionnal score
Missing values can be ignored and conditions can be set for indicators:
# oos_mse weight reduced to 1 when the other
# indicators are "Bad" ou "Severe"
condition1 <- list(
indicator = "oos_mse",
conditions = c(
"residuals_independency",
"residuals_homoskedasticity",
"residuals_normality"
),
conditions_modalities = c("Bad", "Severe")
)
BQ <- compute_score(BQ,
n_contrib_score = 5,
conditional_indicator = list(condition1),
na.rm = TRUE
)Customize the score computation
Practical steps if you want to customize the score computation (see package documentation in R)
select your indicators of interest
adjust “good”, “bad”…threshold in JD+ GUI if necessary
by default good=0, uncertain=1, bad or severe=3
change this grading system and/or the weights directly in the package functions
rebuild your package
List of exportable series
Some available output series will be different when using X13-ARIMA or TRAMO-SEATS.
List of exportable diagnostics and parameters
Some parameters and available diagnostics will be different when using X13-ARIMA or TRAMO-SEATS.
options(
default_matrix_item =
c(
"period", "span.start", "span.end", "span.n", "span.missing",
"espan.start", "espan.end", "espan.n", "log", "adjust", "regression.lp",
"regression.ntd", "regression.nmh", "regression.td-derived",
"regression.td-ftest", "regression.easter", "regression.nout",
"regression.noutao", "regression.noutls", "regression.nouttc",
"regression.noutso", "regression.td(*):4", "regression.out(*)",
"regression.user(*)", "likelihood.neffectiveobs", "likelihood.np",
"likelihood.logvalue", "likelihood.adjustedlogvalue", "likelihood.ssqerr",
"likelihood.aic", "likelihood.aicc", "likelihood.bic", "likelihood.bicc",
"residuals.ser", "residuals.ser-ml", "residuals.mean", "residuals.skewness:3",
"residuals.kurtosis:3", "residuals.dh", "residuals.lb", "residuals.lb2:3",
"residuals.seaslb", "residuals.bp", "residuals.bp2", "residuals.seasbp",
"residuals.nudruns", "residuals.ludruns", "residuals.nruns",
"residuals.lruns", "arima", "arima.mean", "arima.p", "arima.d",
"arima.q", "arima.bp", "arima.bd", "arima.bq", "arima.phi(*)",
"arima.bphi(*)", "arima.th(*)", "arima.bth(*)", "decomposition.seasonality",
"decomposition.parameters_cutoff", "decomposition.model_changed",
"decomposition.tvar-estimator", "decomposition.tvar-estimate",
"decomposition.tvar-pvalue", "decomposition.savar-estimator",
"decomposition.savar-estimate", "decomposition.savar-pvalue",
"decomposition.svar-estimator", "decomposition.svar-estimate",
"decomposition.svar-pvalue", "decomposition.ivar-estimator",
"decomposition.ivar-estimate", "decomposition.ivar-pvalue", "decomposition.tscorr-estimator",
"decomposition.tscorr-estimate", "decomposition.tscorr-pvalue",
"decomposition.ticorr-estimator", "decomposition.ticorr-estimate",
"decomposition.ticorr-pvalue", "decomposition.sicorr-estimator",
"decomposition.sicorr-estimate", "decomposition.sicorr-pvalue",
"decomposition.ar_root(*)", "decomposition.ma_root(*)", "method",
"variancedecomposition.cycle", "variancedecomposition.seasonality",
"variancedecomposition.irregular", "variancedecomposition.tdh",
"variancedecomposition.others", "variancedecomposition.total",
"diagnostics.logstat", "diagnostics.levelstat", "diagnostics.fcast-insample-mean",
"diagnostics.fcast-outsample-mean", "diagnostics.fcast-outsample-variance",
"diagnostics.seas-lin-f", "diagnostics.seas-lin-qs", "diagnostics.seas-lin-kw",
"diagnostics.seas-lin-friedman", "diagnostics.seas-lin-periodogram",
"diagnostics.seas-lin-spectralpeaks", "diagnostics.seas-si-combined",
"diagnostics.seas-si-evolutive", "diagnostics.seas-si-stable",
"diagnostics.seas-res-f", "diagnostics.seas-res-qs", "diagnostics.seas-res-kw",
"diagnostics.seas-res-friedman", "diagnostics.seas-res-periodogram",
"diagnostics.seas-res-spectralpeaks", "diagnostics.seas-res-combined",
"diagnostics.seas-res-combined3", "diagnostics.seas-res-evolutive",
"diagnostics.seas-res-stable", "diagnostics.seas-i-f", "diagnostics.seas-i-qs",
"diagnostics.seas-i-kw", "diagnostics.seas-i-periodogram", "diagnostics.seas-i-spectralpeaks",
"diagnostics.seas-i-combined", "diagnostics.seas-i-combined3",
"diagnostics.seas-i-evolutive", "diagnostics.seas-i-stable",
"diagnostics.seas-sa-f", "diagnostics.seas-sa-qs", "diagnostics.seas-sa-kw",
"diagnostics.seas-sa-friedman", "diagnostics.seas-sa-periodogram",
"diagnostics.seas-sa-spectralpeaks", "diagnostics.seas-sa-combined",
"diagnostics.seas-sa-combined3", "diagnostics.seas-sa-evolutive",
"diagnostics.seas-sa-stable", "diagnostics.seas-sa-ac1", "diagnostics.td-sa-all",
"diagnostics.td-sa-last", "diagnostics.td-i-all", "diagnostics.td-i-last",
"diagnostics.td-res-all", "diagnostics.td-res-last", "diagnostics.ic-ratio-henderson",
"diagnostics.ic-ratio", "diagnostics.msr-global", "diagnostics.msr(*)",
"decomposition.trendfilter", "decomposition.seasfilter", "m-statistics.m1",
"m-statistics.m2", "m-statistics.m3", "m-statistics.m4", "m-statistics.m5",
"m-statistics.m6", "m-statistics.m7", "m-statistics.m8", "m-statistics.m9",
"m-statistics.m10", "m-statistics.m11", "m-statistics.q", "m-statistics.q-m2",
"diagnostics.basic checks.definition:2", "diagnostics.basic checks.annual totals:2",
"diagnostics.visual spectral analysis.spectral seas peaks", "diagnostics.visual spectral analysis.spectral td peaks",
"diagnostics.regarima residuals.normality:2", "diagnostics.regarima residuals.independence:2",
"diagnostics.regarima residuals.spectral td peaks:2", "diagnostics.regarima residuals.spectral seas peaks:2",
"diagnostics.outliers.number of outliers:2", "diagnostics.out-of-sample.mean:2",
"diagnostics.out-of-sample.mse:2", "diagnostics.m-statistics.q:2",
"diagnostics.m-statistics.q-m2:2", "diagnostics.seats.seas variance:2",
"diagnostics.seats.irregular variance:2", "diagnostics.seats.seas/irr cross-correlation:2",
"diagnostics.residual seasonality tests.qs test on sa:2", "diagnostics.residual seasonality tests.qs test on i:2",
"diagnostics.residual seasonality tests.f-test on sa (seasonal dummies):2",
"diagnostics.residual seasonality tests.f-test on i (seasonal dummies):2",
"diagnostics.combined seasonality test.combined seasonality test on sa:2",
"diagnostics.combined seasonality test.combined seasonality test on sa (last 3 years):2",
"diagnostics.combined seasonality test.combined seasonality test on irregular:2",
"diagnostics.residual trading days tests.f-test on sa (td):2",
"diagnostics.residual trading days tests.f-test on i (td):2",
"diagnostics.quality"
)
)