Toulouse president Olivier Sadran has warned he is "very, very worried" about the club's future amid falling attendances.