If you hate your job, the money becomes just numbers in an account. I gave up a job in London earning around $250k in my late 20s as like you I hated every single day (that was in a corporate finance role). Slightly different driver to you though, I really hated the job and living in London and decided I just had to get out. Ended up taking a less pressured job on more like $150k in a smaller city (Bristol), living in the country and it was the best thing I ever did. Now I still own a country home in the UK and perhaps because it was evident to my bosses that I was happy in my work, I've been promoted a few times and am on a bit more than I was in London years back and working at the head office in Paris, France. I'm happy I made the move as now I have a job I love, in a place I love and I look back on my London life and realise I was even more miserable than I thought at the time.
Also I have friends who've done kind similar things to what you plan: one friend started a small IT company in his mid 20s, he's now worth millions as he sold that company and has started a new one, another friend started a property investment fund buying German property (he quit a $800k per year investment banking director role for that) and he's also a multi-millionaire, another friend quit a succesful career in finance to retrain as an architect and now runs his own small practice - he is NOT a multi millionaire but the guy lives a comfortable life in rural England and works about 20 hours a week, another friend gave up a job in finance at a large hedge fund manager to become a property developer in eastern europe, he was a multi-millionaire but his shit has firmly hit the fan now that the bubble has burst so I think he's struggling now.
Good luck if you decide to go for it, many people do, some succeed, some don't but I guess if your heart says go for it then why not. You only live once.