Everything seems to be ok.
As far as i know, spdif is just ac3 raw data, sent to the sound card as a stereo 16bit 48k signal, with just a flag saying "it's not a stereo signal, it need to be decoded".
(and the option "output spdif as pcm" remove this flag, so the sound card don't decode anything, and play weird noises in the speakers
So in my plugin, i do just that :
- i tell ac3filter to use spdif (so encoding to ac3, the flag, ...)
- i sent data encoded by ac3filter directly to directsound, as an SPDIF signal, using ac3filter
- i send dummy data in the winamp ouput plugin instead of the real signal (it produce weird noises)
*i don't use the output plugin for spdif, as it doesn't work at all (for me at least)
as i'm writing, i noticed that i play two things at the same time, the real signal to directsound directly, and a dummy signal to the output plugin (which also play with directsound). And now, i think it is REALLY a bad thing, don't know why i didn't see that before.
That might explain the weird part of your tests too.
When spdif is used in my plugin, the output plugin of winamp plays only a flat wav signal. If you set the winamp output plugin to SPDIF device, the device tries to decode the wav signal as if it was an ac3 encoded signal. And of course, it fails, and .... i have no idea, but nothing not good happens
And as there is only one SPDIF output possible at the same time, one signal take over the over one, unfortunately, the bad one (from the output plugin) take over the correct one in your case.
So if i'm right (don't know, i'm just guessing), if you use spdif, do not set the ouput plugin to SPDIF, and on my side, i should not use the output plugin at all