Let me start by saying that I'm a PC gamer, exclusively. I have never had a use (nor do I ever intend to) for consoles for gaming. I purchased the Xbox One because I wanted to integrate my media and other experiences with voice and gesture control.
So, I realize this is a 1.0 product, so there are certain aspects I'm not going to get wound up about. I love Windows Phone. I started with a Samsung Focus, then went to the Nokia Lumia 900 (which I gave to a friend interested in trying it out), next to the Lumia 920 (which I still have and keep updated as a spare) and now to the Lumia 1020 (by far the best smartphone out there, in my opinion). I followed Windows 8 starting with the original Preview all the way through 8.1 and I love it as well, although it's not nearly as mature as Windows Phone and really needs to catch up. So, I like the Xbox One interface as well...although IT is not even as mature as Windows 8.1.
I love being able to walk into the room and simply say "Xbox ON" and my home theater system comes to life, with the Xbox saying "Hi, Robert!". I love that it recognizes my wife as well and allows us to each have our own customized start screen (I'm going keep using the Windows 8/Windows Phone vernacular). I love the gesture controls as well, and am getting better at muscle memory and refining my movements. And, for the most part, I enjoy many of the apps that are available.
Now, the downside and wishlist. The voice and the gesture experiences aren't quite consistent enough, in my opinion. I shouldn't have to use the controller at all, as I see it. I should be able to use voice recognition even for entering text instead of an onscreen keyboard that isn't even consistent across different apps. If I bring up a search box in an app or settings I should be able to speak the text. I'm hoping that with the update rumored to bring Cortana we will see that capability. Also, this may be the improved Kinect, but I'm guessing it isn't as good as the updated one for Windows. It has a hard time properly reading my movements if I get closer than four feet. If I'm wearing my glasses instead of contacts it doesn't recognize me (same for my wife).
I would like to see a LOT more of the apps from Windows 8 be available on Xbox One and, in fact, I would like to be able to see a profile sync option. I want to be able to say, "Xbox MAIL", Xbox CALENDAR" and so on. I would like to see more two-handed gestures recognized. As it is it takes some tricky movement to get the only two-handed gesture that is currently recognized to actually work. Also: HOME AUTOMATION. If the Xbox One is supposed to be the ONE, for the future I should be able to integrate more automation capabilities. If that's in the form of apps, that's fine. For example, I have spoken to our home security/automation company, Vivint, about integrating with the Xbox One. They were actually intrigued by the idea. As I see it, there's no reason why, using the Xbox One, I couldn't interact with my home security and automation using voice and gesture as well. And if Cortana is added and dramatically expands the ability to give instruction, THIS is how I see the Xbox One really leaving the PS4 in the cheap seats. The market is no longer in catering to gamers. It's to the rest of us who will never use a console for gaming...but are open to other possibilities.