Oh, Scarlet Witch... When your superpower is called a "hex" you probably have trouble making friends. This would seem especially true when you manage to become an Avenger - a notoriously touchy and fragile group. And judging from the info coming out preceding Avengers 2: Age of Ultron, it looks like she's not going to end up being on Thor's Christmas list when all is said and done.
Scarlet Witch's powers have a history of doing everything from getting into people's heads, or Ultron's circuitry (!), to completely changing the fabric of reality. One thing she's pretty well known for is offering glimpses of the future - something that's not so bright when it comes to Asgard.
We know that the next Thor stand-alone film is Thor 3: Ragnarok, coming in 2017, and anyone with a bit of background in Norse mythology knows that this means some dark times ahead for Asgard. Also confirmed is that some of Thor's buddies, Loki (Tom Hiddleston) and Heimdall (Idris Elba), will be making appearances along with various Asgardians being listed in the cast so we can count on a trip to Asgard in [The Avengers: Age Of Ultron](movie:293035).
Marvel Comics history, as well as Norse mythology, tells us that Ragnarok is the end of times for Asgard and pretty much an inevitability. The story plays out a bit like the Biblical flood with Thor standing in for Noah - finding and saving what Asgardians he can and eventually recreating Asgard on Earth.
From what bits we've gathered from trailers and Marvel's Phase 3 announcement video, the pieces seem to be falling into place for Asgard and Thor. If this cataclysmic event is what the Scarlet Witch gives Thor a glimpse of, it makes perfect sense as to why he decides to take off and leave the Avengers hanging in their fight against Ultron.
Family first, as they say.