Hammer hooks and sear engagements are tiny little things that are designed to operate unimpeded. Note that the sear and disco are just sort of floating around loosely held in place by the sear spring. They need to be able to jiggle when they need to jiggle, and they need to be where the hammer expects them to be when the gun is cocked. Note that "Assembly Lube" is just heavy oil, but it can hold parts in place pretty darn well. Grease is like glue compared to assembly lube.
You are not doing much shooting, so I'm not surprised you aren't seeing much dirt. I shoot at least 300 rounds every week - I do a total teardown and cleaning every 1200 rounds. I have seen lots of solid caked residue with all lubricants except Ballistol. With Ballistol I get soft black gunk that washes right out with... More Ballistol. Hoppe's with a little 0-20 synthetic motor oil added would probably behave about the same.
Grease doesn't attract dust and debris. Grease just sits there and the gun fills up with primer and powder residue (and if you shoot lead, lubricant residue) and whatever falls on the grease stays in the grease and gets churned around in the grease, and reacts with the grease to form caked sludge. Dust and dirt in the air get added to the mix. So do textile fibers if you carry.