I avoided proper loungewear for years, convinced it was just an expensive way of describing pyjamas to people who could afford better pyjamas. What changed my mind was a stretch of being ill for a week and living in the same soft set the whole time, and realising it didn't look like I'd given up on life the way my old joggers did.
The useful thing about a good loungewear set is that it doesn't stay in the house. A decent wide leg jogger goes under a jumper for the school run, a relaxed t-shirt or hoodie doubles as something you'd actually answer the door in, and a soft zip jacket works over gym kit or under a coat on a cold platform. It sits in that gap between pyjamas and real clothes, which sounds like a small gap until you've lived in it. I wear mine for working from home, for the drive to visit my mother, for the two hours after a long shift where I can't face proper trousers but still have to be seen in public.
Where it falls apart is almost always the fabric. Cheap loungewear pills within a few washes, the cuffs go slack, and the colour that looked like a soft oatmeal in the photo turns grey and bobbled by week six. A better one uses a heavier cotton or a cotton and modal mix with a bit of weight to it, the kind that holds its shape through a hot wash rather than coming out thinner and see through. Check the seams, especially round the crotch and underarm on joggers, because that's where the cheap stuff splits first. Decent elastic in the waistband matters more than anything else in the set, since a waistband that's gone by October is a waistband that's gone for good. Good loungewear should feel slightly structured even when it's soft, not like a sack you've been poured into.
If you're starting from nothing, go for a plain set in a mid grey or navy rather than anything pale or heavily printed, because it hides the inevitable mark from tea or dinner and doesn't date the way a slogan or a seasonal print does. A simple jogger and sweatshirt combo in a proper cotton blend will outlast almost anything trend led, and you can build around it with whatever t-shirt or jacket you already own.
Mine has been through more hot washes than I'd like to admit and still looks like loungewear rather than laundry. That's really all I ask of it.