Part of it is that, let's face it - he was one of the last of a breed of actors who was able to play, well, regular people.
If you just limited him to The Odd Couple, you would have a pretty damn good reason to miss him. For some, it's Quincy, ME, one of the last of the great 1970s/1980s crime dramas.
But go back further, and Klugman was one of the architects of the "Golden Age of Television", appearing in many early live television plays. (He's also tied with Burgess Meredith for the most appearances on The Twilight Zone, and all of Klugman's appearances are top notch). In his later life, he became an aggressive activist for tobacco prevention after dealing with a serious bout of throat cancer. In every aspect of his life, he was scrappy, abrupt, and argumentative....and we need more actors like him - people willing to take on the image of a scrapper.
For me, I'll celebrate his life by digging out my DVD copy of one of my favorite Klugman films. I read the play in high school, saw the original in college (managed to avoid the remake), but quite honestly, it's one of my favorite Klugman performances. Here's the trailer: