Neighbors upset with plans for stately home