Stephen Colbert’s tenure on The Late Show has been rocky. He has struggled to translate his fake news popularity to a more straightforward late night format, and ratings haven’t been awesome. But more importantly, Colbert struggles to create the kind of viral content networks love to see blowing up on the internet the next morning. His even later late-night counterpart, James Corden, has a viral hit with “Carpool Karaoke”—confession, I really do not like this bit—and Corden has been doing all right building up a late-late audience with his group interview style, modeled after the UK’s The Graham Norton Show.

So Colbert keeps getting upstaged by his fellow network late-nighter, and recently his show has become a revolving door of producers as CBS tries to find the magic combination of people to help him find his feet on The Late Show. Howard Stern even brought up the idea of Corden taking over for Colbert, which Corden flat-out denies, but let’s be honest. It’s never a good sign when people start speculating about stuff like this. Remember Conan O’Brien on The Tonight Show? No one thought that would go the way it did, either.

And now David Letterman, sporting the obligatory retirement beard in an interview with Tom Brokaw, is talking about Colbert and The Late Show and it’s not exactly a ringing endorsement. He doesn’t sh*t on Colbert or anything, but he does say, “I don’t know why they didn’t give my show to a woman. That would have been fine.” The obvious insert here is Samantha Bee, who is f*cking KILLING IT with her show Full Frontal (her piece on the rape kit backlog is scathing). But Bee has a luxury Colbert no longer does, and that is that she can be political. Colbert, anchoring a mainstream late night show on one of the big three networks, can’t be as political as he was on Comedy Central.

It’s really holding him back. Political satire is Colbert’s bread and butter. He openly looks bored—or worse, combative—when forced to interview celebrities about movies he’s clearly not seen, and he seems pained when doing the kind of material designed to create viral buzz. Colbert is SO talented, and there’s no question he can hold down The Late Show—if The Late Show will just let him be Stephen Colbert. I hope they give him through the fall, at least. We’re about to have the most balls-out crazy election in the history of the United States and it would be a shame to bench one of the best satirists we have before said craziness really begins. (Oh you thought it already started? Please. This is the warm up.)

Ideally, CBS will keep Colbert and let him come back in the fall with a refreshed show more focused on politics than celebrity interviews, but that means changing the status quo. The late night format has been standardized for decades. Most of the experimenting with that format has gone on during the late-late shows, or on basic cable where there is less ratings pressure. As long as CBS cares about maintaining the status quo as determined by Johnny Carson, I don’t think Stephen Colbert will last much longer as their Late Show host. People are already asking about potential replacements, and David Letterman himself is openly questioning why the show didn’t go to someone else. Is this the writing on the wall?