I have a pole barn building with a basement and I cannot get a fixed rate refinancing my mortgage. Fannie mae wont touch it because of metal siding, freddie mac wont touch it because i live in a rural area and there are not any sales of these homes in the surrounding fifteen miles. Am i going to find any lenders that will give me an in house loan that will give me a fixed rate?
Did you try a small, community bank? Usually they will be more open to your situation but probably have more stringent requirements like larger downpayments, higher credit scores, more collateral, etc.
Another option is the Farm Credit System. They cater to people like yourself who don't fit the standard mortgage profile of Fannie & Freddie.
Hope this helps.