You seem to be suggesting that 2000sf is small, but to me that seems huge. I live in a 1200sf house in Minneapolis with my wife. When we move, we'll probably be looking at houses slightly larger (so we can fit a grand piano), but we definitely don't need 2000sf, much less 4000!
It's amazing how big newer housing is, and in location that aren't that far from older, smaller housing that costs only slightly less.
For example, in my area, there's a sizable development that was built during the 50s and 60s. The average home is under 2000 sq ft and pretty nice for the time (lots of split levels). 3 or 4 bedroom etc. Lots are .25 to .5 acre. But being older homes you can't, for example, fit a king bed in any of those homes. Or modern bathrooms, where 2 people can get ready in the morning at the same time don't exist there. You might be able to cram a grand piano into one of the larger basements, with nothing else, or replace the family room with it in a tight squeeze.
The crime rate is up in this area, and last year a woman was found chopped up and stuffed into a set of suitcases that were dumped behind some dumpsters at the nearest McDonalds. Schools are old and run down, they just shut down the last remaining library, there's 1 public pool and though shopping is plentiful, it's all a 15-20 minute drive away.
The next area over was built in the early 2000s during the boom and the town houses start at 2400 sq ft. It's almost insane. A smaller single family with a finished basement starts at around 3200 sq ft and the average size is pushing up on 4000 sq ft. Lots are about .25 acre.
The difference in price between the two locations is $0 for the town houses and about $100-150 for the smaller single-family.
I'd also add that the schools in the newer area are all new and very nice, shopping is better, there's convenient walking to local town centers, public transport to the metro system and the nearest cities. The neighborhoods are all fiber internet have in-neighborhood police/fire "public safety centers", several neighborhood pools and plentiful community activities -- among other amenities.
So all things being equal, you may not be happier with the larger home, but it doesn't hurt, and the neighborhood is a lot nicer.