So I decided to build some Hobbit houses into the side of a hill. The tunnel doors work as expected since they can be placed into a mined hole. The problem is that windows can’t be inserted in earth in the same way. This may just reflect that windows can’t be built into an existing hole regardless of the material of the “walls.”
I figured out a workaround, but it requires prior planning. You can insert windows if you mine out the space where the windows go but leave a layer of earth behind them. Then mine out the wall of earth behind them. If you try to build a wall behind the hole and insert a window, then remove the wall, it takes the window with it.