Both me and my friend are experiencing this problem.
We have been in chats together for some periods of time so its not like it has never worked. However one time it dropped me/him out of the party with no warning.
Other times we can join the party, but neither of us can turn the party chat on.
I am with Sky broadband (Only ADSL unfortunately) and my friend has BT fibre.
We have both done power cycles.
I have seen others on these forums mentioning this however i also saw some xbox reps saying that each issue should have its own thread. so here it is.
These are the stats direct from my xbox one (did 2 tests)
2.2 -3.4mb download
0.9- 1.4mb upload
Nat type: Moderate.
Judging by what others have said the NAT type seems to be an issue with the xbox side of things (oddly) so the fact i have Moderate Nat shouldnt affect me as well even with moderate i can still connect with people who have open/moderate nat types. Plus i believe i have had a moderate nat type for some time and i have always been fine on the 360, more often than not hosting games for me and my friends. so my network can, and has handled chat easily before.