There does not appear to be a geographical hinderance to shifting the alignment of the railway by a few feet so guess it was a stubborn refusal to abandon the property at the time the line was built (1860 according to HE above).
The line was still being built (or, at least, was unfinished) come the 1861 census - I've edited a book on the history of Colwall - but as I recall, the railway, and especially the big tunnels at Colwall and Ledbury, along with the large viaduct at the latter, was really quite civils intensive and costly, so they probably lacked the extra budget to argue the case. It's not easy to see on Google Streetview, but the realignment seems to have been in the middle of the formation!
Adam