Hi and thanks for reaching Microsoft Xbox gamer's community. My name is Josef.
I'm an Independent Advisor and a Microsoft Xbox user like you.
Hope you're doing great today despite this Brandon
I can assist you with some things but if the issue persist you may need to reach Mojang support
https://help.minecraft.net/hc/en-us/requests/newLet's try to isolate the problem a bit more, for this:
Try enabling LAN mode in the world that you want to play with your brother.
Also, make sure that Minecraft has enabled network permissions if necessary.
For this go to settings > privacy and security > local network > enable the toggle if Minecraft appears.
If the issue persist, this can be a bit dumb, but may I know if you tried removing your brother from your friends list and then re-add it? (Ask him to do the same)
**Also, make sure that Minecraft is up to date