In 1864 Fanny and five-year-old Bell, travelled by ship from New York to Panama, crossed the isthmus, and then continued up the coast to San Francisco before travelling inland to Nevada. The young family then spent several difficult years in mining camps until resettling in Oakland, California. 

By 1875, however, the marriage had deteriorated and Fanny was unhappy. Despite giving birth to sons, Lloyd (1868) and Hervey (1871), Fanny and Samuel had drifted apart and Fanny left her husband to travel in Europe taking her children with her. 

Attempting to study art with Belle in Antwerp, the small group learned women were not permitted into the school and so they returned to Paris where they enrolled at the Académie Julian. Soon after arriving, Hervey, her youngest, contracted Tuberculosis and died in April 1876.

Distraught, Fanny moved with her children to the French countryside and the small artist-populated town of Grez-sur-Loing near Barbizon. There she met artist, Robert Alan Mowbray Stevenson. That summer, his cousin, Robert Louis Stevenson, was visiting. Fresh from university, Louis was seeking inspiration for his writing and instead found true love. Conflicted for a growing love for another man, Fanny and her children returned to California in 1878. 

A year later, Fanny was convinced there was no salvaging her marriage and wrote to Stevenson. At the end of August 1879 Stevenson arrived on her doorstep in Monterey – looking emaciated. Unwell and with little money to his name, Fanny still chose Stevenson and divorced her husband in December of 1879.