Gilbert's father, Jean Gilbert, born Max Winterfeld, was the son of a Hamburg businessman. Winterfeld struggled to enter the world of the theater, beginning as a conductor and composing various pieces. French vaudeville was in fashion and in 1901, when Robert was one year old, he adopted the penname of Jean Gilbert. The family was poor for years and lived in seedy hotels, his father had to work nights playing piano wherever he could find work. Sometimes it was so cold that Robert's mother, Rosa, had to keep warm in bed while sewing hats to earn a little extra money.