Old film where a guy tears his house apart at the end

I remember seeing a film on TV when I was a kid (1980s) that was old at the time, probably 1960s.  It was a comedy where the climax was the main guy getting so fed up with his house that he starts to rip it down with his bare hands.  It might have starred James Stewart.  I think it was in color.  I seem to remember that he was wearing a red shirt with a white collar, or possibly a red sweater over a white dress shirt.

For the longest time, I thought it was Mr. Blandings Builds His Dream House, which stars Cary Grant, not James Stewart, and is in black and white. I just finished watching that film, and he does not tear his house apart.  He is happily living there at the end of the film even though much of the comedy involves things costing much more than he expected.  There is some slapstick with the unfinished house, but overall, it's pretty genteel, so it definitely wasn't the film I was thinking of.

I also thought it might have been Mr. Hobbs Takes a Vacation, but I saw that last year, and that wasn't it, either, although I may have seen that one as a kid.