Comparison of breakpoint detection algorithms on the neuroblastoma data

Breakpoint detection error rates are shown in percent for several models trained on the annotated neuroblastoma data.

Test error rates were estimated using leave-one-out cross-validation on annotated breakpoint regions in a database of 575 array CGH profiles with up to 6 visual annotations per profile. For details, read Learning smoothing models of copy number profiles using breakpoint annotations.
The R code that implements each algorithm can be found in the smoothers variable, defined in smoothing-functions.R (older version in bams on CRAN).
This web page was produced using the accuracy.R script. The detailed error counts for each algorithm are in in all.stats.RData (computed by running smooth-one.R for each profile via smoothing-commands.R and then combining the results using make.all.stats.R -- see Using R on the CBIO cluster for a more detailed explanation).
Global test error is used to order the rows of the table and the panels of the plots.
FP = False Positive = the model detects a breakpoint in a region annotated to have none.
FN = False Negative = the model detects no breakpoints in a region annotated to have at least one.

	test error, learning, generalization							train error, fitting
	global			local			Timings	global			local
	errors	FP	FN	errors	FP	FN	seconds	error	FP	FN	error	FP	FN
multiBinSeg	2.2	0.6	11.6	11.1	12.8	7.0	0.59	2.2	0.6	10.1	0.3	0.2	0.5
cghseg.k	2.2	0.6	11.6	11.1	13.0	7.0	1.69	2.2	0.6	10.1	0.3	0.2	0.7
cghseg.k1	2.2	0.6	11.6	11.1	13.0	7.0	2.67	2.2	0.6	10.1	0.3	0.2	0.7
fpop	2.2	0.6	11.6	11.1	13.0	7.0	4.51	2.2	0.6	10.1	0.3	0.2	0.7
pelt.n	2.2	0.6	11.6	11.1	12.9	6.9	5.83	2.2	0.6	10.1	0.2	0.1	0.7
cghseg.klogn	2.4	0.6	13.0	10.9	12.7	7.2	1.73	2.4	0.6	11.3	0.2	0.1	0.7
smuce.penalized	2.4	0.6	13.5	10.7	12.5	6.6	122.74	2.4	0.6	11.7	0.4	0.2	1.2
cghseg.k.var	2.7	1.5	9.8	12.7	14.7	6.2	3.14	2.5	1.1	9.1	0.4	0.3	0.9
cghseg.k.less	3.1	2.2	8.4	8.4	9.8	5.8	2.15	3.1	2.2	7.5	0.4	0.4	0.9
pelt.meanvar.n	3.2	1.5	11.1	11.6	12.9	8.0	10.42	2.7	1.1	10.6	0.5	0.2	1.7
wbs.penalized	3.9	2.0	15.0	11.5	13.3	8.1	3.50	3.5	1.4	14.1	0.5	0.4	1.0
cghseg.k.sqrt.d	3.9	2.3	12.3	12.4	14.6	7.0	2.62	3.5	2.3	9.4	0.3	0.2	0.7
cghseg.k.sqrt.d.var	4.5	3.1	12.5	12.5	14.2	7.9	2.46	3.2	1.7	10.8	0.5	0.4	1.0
cghseg.resid.corr	4.5	2.6	14.3	13.2	14.3	9.3	1.72	3.8	1.6	14.7	0.9	0.7	1.9
pelt.asymptotic	5.5	4.4	11.3	5.8	6.1	6.9	4.54	4.8	4.3	7.2	0.8	0.4	2.8
cghseg.schwarz	6.0	3.7	18.5	11.0	12.5	8.3	1.81	5.3	3.3	15.2	0.5	0.4	0.9
cghseg.k.sqrt	6.1	4.0	14.7	12.0	13.2	8.4	2.93	4.3	2.5	13.4	0.5	0.4	0.9
cghseg.k.sqrt.var	6.4	4.3	16.2	13.6	14.6	9.1	2.35	4.2	1.8	16.2	0.7	0.6	1.6
gada	6.5	4.1	19.8	12.9	14.4	10.4	6.36	4.7	1.8	18.7	0.6	0.5	1.6
flsa.norm	6.7	3.6	18.5	14.8	15.2	10.5	0.08	4.8	2.3	17.5	1.2	0.8	3.0
wbs.th.const	6.8	4.0	22.2	14.8	16.7	8.5	5.34	4.7	2.5	16.1	0.8	0.5	2.6
pelt.diffparam	7.0	4.6	22.1	13.7	15.9	7.0	3.90	6.0	3.3	19.7	1.2	1.3	0.5
pelt.manual	7.0	4.6	22.1	13.7	15.9	7.0	3.85	6.0	3.3	19.7	1.2	1.3	0.5
cghseg.c	7.7	5.7	18.8	13.2	14.5	10.0	1.85	4.2	2.3	13.6	0.6	0.5	1.2
pelt.default	8.0	1.1	44.8	8.0	1.1	44.8	0.19	8.0	1.1	42.2	8.0	1.1	42.2
smuce.q	8.9	5.3	27.5	14.1	14.5	13.2	122.57	5.8	1.4	27.9	0.8	0.5	2.8
wbs.th	9.1	5.7	26.0	12.8	14.4	9.5	5.23	7.8	5.6	18.5	0.5	0.4	0.9
pelt.meanvar	9.3	5.4	30.1	14.1	15.0	10.7	10.80	6.9	3.8	22.5	1.1	0.7	2.6
dnacopy.sd	11.5	7.6	32.2	12.8	10.0	28.8	51.62	11.5	7.3	32.1	2.5	1.5	7.5
glad.haarseg	11.8	12.6	8.0	17.8	20.7	4.1	29.51	11.7	12.4	8.2	9.0	10.5	1.6
glad.lambdabreak	14.1	12.3	23.0	15.8	16.2	14.2	14.44	13.4	10.9	26.0	6.4	7.2	2.1
flsa	16.0	12.7	36.6	14.4	15.9	10.3	0.04	11.3	6.9	33.3	1.3	1.3	1.4
cghseg.lambda	16.8	0.0	100.0	30.5	22.1	65.3	1.84	16.8	0.0	100.0	12.9	3.2	61.4
cghseg.min.abs.diff	17.0	0.3	99.7	21.5	19.8	26.6	1.76	15.8	0.4	92.7	5.1	4.0	10.5
dnacopy.alpha	18.4	21.9	2.6	20.4	24.6	2.0	25.90	18.3	21.4	2.6	17.9	21.2	1.4
glad.MinBkpWeight	25.2	30.0	4.3	25.8	31.2	1.3	40.88	25.2	29.3	4.7	19.7	23.5	0.7
glad.default	27.4	33.3	1.2	27.4	33.3	1.2	1.13	27.4	32.6	1.6	27.4	32.6	1.6
dnacopy.prune	27.9	31.9	17.1	31.1	36.5	10.6	35.17	27.7	30.5	13.9	25.9	30.5	2.8
dnacopy.default	40.5	49.3	0.5	40.5	49.3	0.5	1.78	40.5	48.5	0.7	40.5	48.5	0.7
cghseg.mBIC	40.9	49.4	0.0	40.9	49.4	0.0	1.77	40.8	49.1	0.0	40.8	49.1	0.0
smuce.alpha	42.3	51.4	0.3	53.2	64.6	0.0	47.08	42.2	50.6	0.3	41.8	50.2	0.0
smuce.default	71.0	85.8	0.0	71.0	85.8	0.0	0.75	70.9	85.2	0.0	70.9	85.2	0.0
wbs.default	72.7	87.6	0.9	72.7	87.6	0.9	4.01	72.7	87.1	0.9	72.7	87.1	0.9
cghseg.schwarz.auto	74.3	89.6	0.0	74.3	89.6	0.0	1.69	74.3	89.2	0.0	74.3	89.2	0.0
gada.default	80.5	96.9	0.0	80.5	96.9	0.0	0.23	80.5	96.7	0.0	80.5	96.7	0.0
cghFLasso	80.8	97.2	0.0	80.8	97.2	0.0	0.14	80.9	97.2	0.0	80.9	97.2	0.0
iir.aic	83.2	100.0	0.0	83.2	100.0	0.0	1.81	83.2	100.0	0.0	83.2	100.0	0.0
iir.aicc	83.2	100.0	0.0	83.2	100.0	0.0	2.08	83.2	100.0	0.0	83.2	100.0	0.0
iir.bic	83.2	100.0	0.0	83.2	100.0	0.0	1.77	83.2	100.0	0.0	83.2	100.0	0.0
iir.gcv	83.2	100.0	0.0	83.2	100.0	0.0	1.76	83.2	100.0	0.0	83.2	100.0	0.0

The global training error curves for models with a tuning parameter are plotted below.
The minimum is marked with a horizontal black line and corresponds to the train error of the global model in the table above.
You can check if we have explored a large enough range of the tuning parameter by looking for the characteristic dip and rise of the error curve.

The timings of the algorithms are shown in the scatterplots below.
Each point represents one profile.
The median time to fit the model to a single profile is shown as a horizontal line. This is the quantity displayed in the table above, and is used to order the panels of the plot.

Made using R, ggplot2, and xtable.