The brain serves as the body's primary regulating center for a wide range of processes, making it essential to preserve its well-being and optimal functioning in order to maintain a good quality of life. There are several major brain disorders, including Alzheimer's disease and brain tumors, which significantly impact the lives of affected individuals. Traditional centralized learning methods, such as machine learning and deep learning, have demonstrated improved performance in detecting these diseases. However, a major concern in the present day is the inability of centralized learning to protect the privacy of subjects' data. Therefore, it is crucial to detect and classify diseases while ensuring the privacy of subjects' data. With this in mind, this research focuses on privacy-preserving cross-silo federated learning for the detection and classification of Alzheimer's disease and brain tumors. For classification purposes, this study employed the LeNet5 model with the Adam and SGD optimizer on the dynamics of T1-weighted magnetic resonance imaging data based on privacy-preserving cross-silo federated learning. The model was utilized for the binary classification of Alzheimer's disease and the multi-class classification of brain tumors for different percentage of client participation in federated settings. In the case of Alzheimer's disease classification, this study achieved approximately average 95% accuracy, 95% precision, 95% recall, and 95% F1 score based on cross-silo federated learning with client participation rates of 90% and 100% at each training round. Furthermore, an average accuracy of 94.66%, precision of 95%, recall of 95%, and F1 score of 95% were achieved for brain tumor classification using the SGD optimizer and enabling LeNet5 model. Additionally, it is very challenging to determine the percentage of client participation required to achieve a moderate or better performance in cross-silo federated learning since the result fluctuates but it somewhat clear that about minimum 20% client participation is required.