Chris
I think the foam board should be fine.
The foam is impermeable, I believe, so if you stick it down to the baseboards with a non-contact adhesive, the solvent is going to hang around a while, and that might be a bad thing, so you might want to go for a solventless approach, either D/S tape or contact glue.
When you ballast the track, conversely, the solvent can escape, and I’d guess that a latex or PVA would be fine. I use latex as it doesn’t make the ballast go green.
‘course this is hypothetical, I haven’t done it, you understand! But my foamboard structures on the Greater Windowledge Railway lasted for years.
atb
Simon