On November 4, 1949, a little royal named Prince Charles celebrated his first birthday in the nursery of Clarence House, his parents’ central London home. While his mother, Princess Elizabeth, watched over him, he was joined by other aristocratic toddlers, including his cousin Prince Richard, son of the Duke of Gloucester. His cake, tinged with rum, had one big orange candle.

Seventy-four years later, he is now King Charles III and has returned to his roots by choosing to live in the five-bedroom Clarence House, although his official residence in London is the massive Buckingham Palace—just a half-mile down the mall. For Charles, it seems Clarence House is a true home filled with happy memories, particularly of his beloved grandmother Elizabeth, the Queen Mother. It is here where he finds solace, surrounded by pictures of his family on display in his office, as he undergoes treatment for an undisclosed type of cancer.

Clarence House, more of an aristocratic townhouse than a palace, has long been a sanctuary for members of the British royal family. It began as a shabby set of apartments on the grounds of the St James Palace. The palace, constructed on the site of a hospital for “leprous maidens,” was built by King Henry VIII in the 16^th century. Over the years, junior royals were housed in buildings adjacent to St James Palace.

During the early 19^th century, one suite of apartments near the stable yards became the London home of Prince William Henry, the Duke of Clarence and the future William IV. Clarence was born in 1765, the third son of King George III and Queen Charlotte. According to Clarence House by Christopher Hussey, William spent his life in the navy charming his compatriots with his bluff friendliness. “You may call me William Guelph,” he told them, “for I am nothing more than a sailor like yourselves.”

“Quick without being clever,” Hussey writes, Clarence had “the careless good nature that made him exceedingly popular with sailors. He was also infamous, due to his twenty-year long relationship with the famous stage comedian Mrs. Jordan, with whom he shared a gaggle of doted-upon illegitimate children given the surname FitzClarence.”

According to Hussey, Dorothea Jordan probably lived in the Clarence House apartments before the couple parted ways in 1811. In 1817, Princess Charlotte, the heir to the throne and the only legitimate grandchild of George III, died in childbirth, spurring her elderly uncle to marry. In 1818, Clarence married the “young, amiable, strictly brought up” Adelaide of Saxe-Meiningen.

It was a surprisingly happy marriage. According to Hussey, the once wild duke settled down, exercising in front of an open window in his drawing room, giving up green tea, and only drinking sherry. Now third in line to the throne, he pressed the head of his pompous brother King George IV’s Privy Purse for a new home, writing:

His majesty is fully aware of the inconvenience and unfitness of our present apartments here. They were arranged for me in 1809 when I was a bachelor…since when (now 15 years) nothing has been done to them; and you well know the dirt and unfitness for the Duchess of our present abode. I earnestly request, for the sake of the amiable and excellent duchess, you will, when the king is quite recovered, represent the wretched state and dirt of our apartments.

Funds were found, and famed architect John Nash was hired to design a new London townhome for the Clarences. Built between 1825 and 1827 on the site of the old apartments, it was an airy, stucco classical structure, far from the pomposity of other royal homes. But just because King George IV funded the project, that did not mean there was an enormous amount of brotherly love. “Look at that idiot!” George IV once said of Clarence, according to John Van der Kiste, author of William IV: The Last Hanoverian King of Britain. “They will remember me, if ever he is in my place.”

George IV died on June 26, 1830. When the 64-year-old Clarence was informed he was now king, he retired to his bedroom at Clarence House, since “he had never slept with a queen before.”

The new king had much in common with Clarence House’s current occupant. He was the oldest man to take the throne until King Charles III was crowned in 2023. Clarence, like Charles, who has joked about his encounters with “frustratingly failing fountain pens,” struggled while signing his first set of official papers, exclaiming, “That is a damned bad pen you have given me!”

Most importantly, he also chose to live at Clarence House but agreed to an underground passage between his home and St James for easy access to the grand palace. The new king and queen lived modestly. Per Hussey, in the evening, Queen Adelaide knitted while her husband soaked his arthritic fingers in hot water between signing thousands of documents that his brother had been too lazy to sign.

King William IV died in 1837, and his niece Queen Victoria ascended to the throne. His sister Princess Augusta Sophia briefly lived in Clarence House before her death in 1840.

In 1866, another sailor-prince, Alfred, Duke of Edinburgh and the second son of Queen Victoria, moved into Clarence House. He and his wife, Grand Duchess Maria Alexandrovna, would haphazardly expand and build onto Clarence House. After Edinburgh’s death in 1900, his younger brother Arthur, the Duke of Connaught, moved in and stayed until his death in 1942. The house fell into disrepair with no central heat, primarily fitted with gas fixtures, and with no modern bathrooms.

During World War II, Clarence House became a hub for the British Red Cross. But in 1947, it was again drafted into royal service when it was announced that Clarence House would become the official home of newlyweds Princess Elizabeth and Prince Philip.

According to Elizabeth the Queen by Sally Bedell Smith, Prince Philip oversaw the home’s extensive two years of renovations, adding modern touches like an up-to-date speaker system. Elizabeth offered more practical advice. According to Smith, when people complained of the home’s new paint smell, she replied, “Put a bucket of hay in there and that’ll take it away.”

The little family moved into Clarence House in the summer of 1949. The home was pastel-colored, brightly lit, and cheery, with accents of Elizabeth’s favorite color, aquamarine. “The foremost impression that a visitor now receives is of the charming, friendly, and quite informal home of a great lady,” Hussey writes, “Of sunshine streaming in at large windows, and of many flowers around a shaded lawn outside and in big vases within the rooms.”

Prince Charles was often left at Clarence House with his nursery staff while his mother stayed with Prince Philip, who was stationed in Malta. Charles was frequently visited by his grandmother Elizabeth, who purportedly understood the sensitive, artistic child more than his practical parents. On August 15, 1950, he was joined by his sister, Princess Anne, who was born at Clarence House. Photos show the children in the Clarence House gardens looking secure and content.

But their stay at Clarence House would be short-lived. King George VI died on February 6, 1952. His 25-year-old daughter was now Queen Elizabeth II. According to the writer Hugo Vickers, the new queen did not want to leave her London home, but Prime Minister Winston Churchill convinced her to move to Buckingham Palace.

This meant that her mother, Elizabeth, the Queen Mother, needed to move out of Buckingham Palace. But if her daughter had been loath to leave Clarence House, the Queen Mother was loath to move in. She finally agreed but was infuriated when Parliament complained about the cost of her redecorations. “Perhaps they would like me to retire decently to Kew and run a needlework guild?” she said sarcastically, per Smith. “You must tell them angrily how little has been done and how loathsome it [Clarence House] is.”

According to The Queen Mother by William Shawcross, Group Captain Peter Townsend was put in charge of organizing the move.

The Queen Mother, her exquisite art collection, and her daughter Princess Margaret finally moved into Clarence House in May 1953. The home had been transformed into “the last word in domestic luxury,” according to the Los Angeles Times, far more comfortable than Buckingham Palace, which the paper called dismal and “the coldest house of Europe.”

The romance between It girl Princess Margaret and Townsend would flourish and die in the rooms of Clarence House. After Margaret married Antony Armstrong-Jones in 1960, Clarence House became the famous domain of her dazzling mother, who one guest recalled as “sparkling with diamonds, in pink tulle crinoline” exuding “an excited joy that was almost unqueenly.”

Clarence House became known as a “glorious gin palace,” famous for its dinners, picnics, and cocktail-heavy receptions that were frequently attended by her favorite grandchild Prince Charles. The Queen Mother filled Clarence House parties with an eclectic guest list and could encourage anyone, even clergy, to loosen up.

“I gave a cocktail party for 200 Bishops from overseas – by the time that 8 o’clock came, they were in cracking form!” the Queen Mother wrote Princess Margaret, per Shawcross. “They tucked into all the canapés ‘and tossed down martini after martini.’”

She also had Clarence House kitted out to encourage her favorite vices. “She read Sporting Life every day and she had installed in Clarence House a rather primitive loudspeaker system, such as usually exists only in betting shops, to relay minute-by-minute news from racetracks around the country,” Shawcross writes.

The house also buzzed with her loyal staff, who adored their eccentric boss and the atmosphere at Clarence House, which Shawcross notes was grand, quirky, and friendly. “She accepted foibles in the men and women who served her,” he writes, “that other royal establishments may have considered a little risqué.”

Ruling the roost was the campy and savage William Tallon, a.k.a. “Backstairs Billy,” the Queen Mother’s right-hand steward who claimed his job was “to keep her smiling.” He was in charge of mixing her cocktails and setting the mood, exclaiming, “When the lights go on in Clarence House, it’s showtime.”

But if Clarence House was a delightful refuge for many, including Prince Charles, it was a prison-like, foreboding place for his future wife. After Charles and Lady Diana Spencer’s engagement was announced in February 1981, the 19-year-old innocent was moved into Clarence House. “Nobody there to welcome me,” she recalled in Diana: Her True Story in Her Own Words by Andrew Morton. “It was like going into a hotel.”

Once she got to her new temporary home, Diana was immediately confronted with her fiancé’s past and future. “At Clarence House I remember being woken in the morning by a very sweet elderly lady who brought in all the papers about the engagement and put them on my bed,” she recalled. “When I arrived at Clarence House there was a letter on my bed from Camilla.”

Three days later, Diana was moved to Buckingham Palace. But the night before her wedding, the Queen Mother invited Diana to spend the night at Clarence House. The Queen Mother was reportedly wary of Diana’s suitability for her beloved grandson and Diana was equally wary of her. That night Diana dined alone with her sister Jane Fellowes and had a severe bout of bulimia.

“They put me in a bedroom overlooking the Mall, which meant I didn’t get any sleep,” she recalled, per Morton. “I was very, very calm, deathly calm. I felt I was a lamb to the slaughter. I knew it and couldn’t do anything about it. My last night of freedom with Jane at Clarence House.”

On July 29, 1981, Clarence House was a beehive of activity as Diana was dressed and made-up as courtiers fluttered in and out. Her proud father, Earl Spencer, walked her down the stairs and said, “Darling, I’m so proud of you.” Out they went to the carriage, Diana bravely meeting her destiny.

When the Queen Mother died in 2002 at the age of 101, Charles moved into both Clarence House and her Scottish home Birkhall. According to Prince Charles by Bedell Smith, he began an extensive $7 million overhaul of Clarence House (paid for by public funds). He personally paid $2.5 million for designer Robert Kime’s interior decoration, per Smith. “He gave Robert Kime two directives,” she writes. “Preserve the essence of the Queen Mother and use as many vintage fabrics as possible.”

He honored his grandmother in other ways. According to The Elements of Organic Gardening by the then Prince Charles and Stephanie Donaldson, between 2004 and 2005, he planted a garden in the Queen Mother’s honor, which his office overlooks.

One day he and his friend Andrés Duany were walking in the Clarence House Gardens. “Over the porte cochère was this slightly opened window,” he told Smith. “And there, from this beautiful neoclassical house, was a little clear plastic hose coming out the window.” When Duany asked Charles what it was, he replied, “I empty my bathtub with a hand pump to water the plants.”

Now the home of a king and queen, Clarence House is the location of many important receptions and parties, and the everyday business of the firm. “King Charles III and Queen Consort Camilla appear to have been heavily influenced by the maximalist style,” Deepa Mehta-Sagar, the founder of Area Decor LLC, told the Daily Express. “On a walk-through, you’ll find that the home is packed with artworks, historic objects, and memorabilia collected from across the globe. Added to this is gilded detailing on cabinets and bannisters. The opulence reflects the storied royal history.”

But it is also a cozy family home. There is a preponderance of family photographs everywhere, and Queen Camilla’s upstairs sitting room, according to Smith, is “crammed with books and knitting, paintings waiting to be hung” and “too much furniture.”

At night, the king and queen often read side by side, surrounded by mementos of loved ones past and present. The palaces are for pomp and tourists, but Clarence House is a true home.