Add a test to avoid regressions. Fixes #3412
that's all this commit does. further commits will fix cli flags and such. see #2221