The purpose of this study is to develop a new VR disaster experience environment construction system with the function which automatically recognizes real world objects’ shape and materials in captured RGB-D images and makes their behaviors in a disaster simulation real. In this system, objects’ materials are recognized by material recognition with machine learning. With the result of material recognition, this system calculates feature values which are values to decide objects’ behaviors in the disaster experience. Feature values consist of proportion of material, flammability and density to decide the sound of collision, how easily objects burn in the fire and whether objects float in water in the flood respectively. An evaluation experiment was conducted to examine whether material recognition improves a sense of reality and fear toward VR disaster experience. The evaluation of the sense of reality toward where fire occurred in the fire experience with material recognition tended to be higher.