Cyrus, a warmly received selection at this year’s San Francisco Film Festival, is a comedy that aims to make audiences laugh but seems willing to settle for making them cringe. But if you can stomach its enfant terrible – a selfish, shamelessly manipulative man-child, desperate to sabotage his mother’s latest romance – you might appreciate the lighter side of his Oedipal obsession.

Because Cyrus is played by Jonah Hill, whose blunt sarcasm and spastic tirades seemed perfectly suited to broader comedies like Superbad (2007) and this month’s Get Him to the Greek, the understated depth he brings to this 21-year-old menace will catch some by surprise. Here, Hill is less flamboyant than simply unnerving, his dark side only partly hidden behind a veneer of faux hospitality.

When Hill liberates Cyrus from the disingenuous small talk and wan pleasantries that are his stock in trade, a different character emerges – a wounded, perilously dysfunctional child, struggling to protect his home from an unwelcome interloper. That man is John, a lonely divorcé played by John C. Reilly, who recognizes Cyrus almost immediately for what he is, and keeps his guard up.

Caught in the middle of this uncomfortable rivalry is Molly (Marisa Tomei), whose intimacy with her son that might help explain his unhealthy attachment. Whether they’re cuddling in the park or sharing the bathroom, Molly and Cyrus are eerily inseparable; their lives revolve around each other, and adding a third party to the mix throws their orbits out of whack.

If that makes Cyrus sound like a natural follow-up to Step Brothers, the 2008 comedy in which Reilly and an aggressively dimwitted Will Ferrell took sibling rivalry to the outer limits of absurdity, it’s really not. Superficial similarities exist – both movies follow two men, each conspiring to lay the other low – but Cyrus deals in real people, not cartoonish buffoons.

That doesn’t stop mumblecore pioneers Mark and Jay Duplass, who wrote and directed the movie, from finding the dark humor in Cyrus and John’s fight for Molly’s attentions. Cyrus is laughably intense; in those rare moments when he tries to relax, the strain is almost palpable. John isn’t so different. Too long a bachelor, he tiptoes around Molly, terrified of smothering her, but his neuroses betray him at every turn.

As they did in the carefully observed road comedy The Puffy Chair (2005) and the horror parody Baghead (2008), the Duplass brothers rely on verité-style technique to give Cyrus a documentary feel. Their habit of zooming in on their subjects, rapidly and without warning, serves to heighten our discomfort as we see them at their most vulnerable. When the movie is most affecting, laughs mitigate the tension without entirely relieving it.

The performances in Cyrus feel as intimate as everything else in Cyrus, Molly and John’s increasingly claustrophobic universe, perhaps because the dialogue is almost entirely improvised and the actors were entrusted with defining their characters in their own words. It’s a choice that has served the Duplass brothers well in past productions, and here, with a gifted cast, it works again.

What doesn’t work as well is the movie’s ending, complete with self-realizations that seem too tidy by half. In Cyrus, the Duplass brothers have crafted a delightfully unsettling comedy that plays on insecurities and the anger they breed. That, finally, they quell the fire that makes Cyrus larger and scarier than life doesn’t really feel like a cop-out, or even a Hollywood ending, but it doesn’t ring as true as all that precedes it.