Hayashibara Museum of Art is located in the area of the inner moat of Okayama Castle. This area was a part of Ninomaru annex of the castle, where there was a guesthouse during the Edo period. Since the Meiji era, Lord Ikeda used it as his administrative headquarters. On this site, Hayashibara Museum of Art was inaugurated on October 1, 1964, as per the wishes of the late Hayashibara Ichiro (1908 -1961), a business person of Okayama.

Hayashibara Ichiro inherited the company that was established by his grandfather, and their specialization was the manufacturing of starch syrup. He later succeeded in making the company one of the biggest Japanese firms of its kind. Alongside his professional responsibilities, he showed an active interest in traditional artwork. Throughout his school days, he devoted a significant amount of time to the study of antique swords. With this growing passion for the arts, he bought the treasures that had been handed down from the Ikeda family, the feudal Lord of Bizen Province, a part of Okayama Prefecture of today. At the same time, he started to collect Japanese and other oriental art pieces.
Regrettably, he passed away in April, 1961 at the age of 52. Three years later, according to the wishes of Hayashibara Ichiro, the bereaved family founded Hayashibara Museum of Art. The project was then endorsed by Miki Yukiharu, the governor of Okayama at the time.

The building was constructed on 6,300m² premises and was designed by Maekawa Kunio, a renowned architect who was a disciple of Le Corbusier. He created the building that was completely harmonized with the cultural environment around Okayama Castle. The museum houses approximately 9,000 pieces of art composed of swords, armors, pottery, Noh costumes and masks, furniture, paintings, calligraphies and lacquer work. The collection comprises 3 national treasures and 26 masterpieces, classified as important cultural property of Japan.