When it came to rebuilding my own life, the piece of advice I was given, over and over, was to “take it one day at a time”. No long-term plans, no significant life changes. I found that unsatisfactory. While there were clearly some decisions it would have been unwise to make while still bowed over in grief, I didn’t want to spend any more time in limbo than I already had. It helped that Kathryn had told me, in no uncertain terms, that I wasn’t to lose my way once she was gone – however difficult, she wanted me to keep moving forwards. She didn’t want me to settle for life without her: she wanted me to make my life anew.

Most of us, most of the time, walk the path of least resistance. I certainly had: though I didn’t particularly like where I lived in the outskirts of west London, or earning my living by editing business reports, it was comfortable enough. But after Kathryn died, my home was no longer my home, and my future no longer my future. I certainly don’t recommend bereavement as a way of hitting the reset button, but it did give me the opportunity and motivation to rethink my life from scratch. It forced me to consider all the possible ways in which I might reconstruct it. At least I still had possibilities to explore.

I started trying on different lives for size: the rural hermit, the man-about-town, the perpetual nomad. A lifelong urbanite, I spent much time tramping in wilds and woods. Without reasons to stay home, I spent my days at galleries and nights at gigs, pursuing moments of escape that rarely came. I’d always travelled as much as I could, but I started crossing destinations off my bucket list in short order. And I started two daily blogs, one for friends and one for the world, to write myself into the future: mournings, imaginings and beginnings.

Some months in, a well-meaning friend asked if I was still on medication. It had never occurred to me to take any. Ditto counselling: I was gently nudged towards it but gently nudged myself away again. I found a support group for young widowers, but I’d never been the kind of person who’d join any club that would accept me. Many in my position observed birthdays, anniversaries, holidays; after the first year, I made the difficult decision to stop observing them. I didn’t want the rest of my life to run to an out-of-date schedule.

It gradually dawned on me that my approach wasn’t entirely typical. I wondered if I was in denial. Or perhaps just an emotionless brute. It didn’t feel as though either of those was true: I was by no means happy or normal during my period of mourning, I just never doubted, even on my darkest days, that better times lay ahead – if I only worked towards them. Initially without really thinking about it, and later more deliberately, I cultivated the idea that the future would be bright. Eventually, I realised that I’d chosen to identify as an optimist.

That was somewhat perplexing. As a trained scientist and a journalist, I was supposedly a hardened critical thinker, committed to solid evidence and rational argument. While I knew, and had been told, that I tended to expect the best out of life, I’d presumed that was because I actually had led a pretty charmed life. To still expect that, after the events of the previous year, felt as if I had given myself over to irrationality: the side of me that believed was winning out over the side that reasoned.

My impression of optimism was that it amounted to nothing more than a belief, and that to place any weight on it was fundamentally silly and potentially irresponsible. Calling yourself an optimist seemed like admitting that you just didn’t want to think very hard about the future and its challenges. But at the same time, what was the alternative? The usual defence of pessimism is that a pessimist is never disappointed, and can only ever be pleasantly surprised. That seemed a needlessly defensive, almost cowardly stance. And professed “realism” seemed to me to be fence-sitting, a cynical excuse to avoid engaging with the possibility that the world could be better than it is today.

I couldn’t see how either of those worldviews would propel you through life. Why would you even bother to get up in the morning?

Once I started to think about it, optimism seemed like the only stance worth taking. At least expecting more out of life primed you to get more out of life, or so it seemed. But I wanted to practise a kind of optimism for which I could articulate a defence that amounted to more than just belief. I wanted to find a way of being an optimist that might actually help make the world better, rather than just assuming it somehow would be.

So I began to investigate what form that pragmatic, well-reasoned version of optimism might actually take. And what I learned was that optimism, despite my earlier assumptions, isn’t necessarily the product of naivety. It isn’t an indulgence that we can only afford when times are good. It’s a resource we can tap into when the going gets tough – and then it can make the difference between life and death.

It was a sickening sensation to feel the decks breaking up under one’s feet,” recounted Ernest Shackleton when, after more than nine months stuck fast in the Antarctic ice, his expedition’s flagship, the Endurance, finally succumbed. And so, on 27 October 1915, the great Anglo-Irish explorer reluctantly ordered his crew to abandon ship – leaving them stranded in perhaps the planet’s most hostile location.

Shackleton’s Imperial Trans-Antarctic Expedition had set off more than a year earlier with the objective of making the first land crossing of the frozen southern continent. Once the Endurance had sunk, however, Shackleton and his men had to make do with the equipment and provisions they had crammed into its three lifeboats. They had no way of communicating with the rest of the world and essentially no hope that anyone would come to rescue them. And yet, 10 months later, Shackleton led every one of his men to safety. How did he and his crew pull through in the face of such incredible odds?

The practical answer is that they spent more than five months camped out on the ice before setting sail in the lifeboats to Elephant Island, which was solid land but had little else going for it. From there, Shackleton took one of the boats and a skeleton crew on a two-week, 800-mile journey through raging seas to South Georgia. On 20 May 1916, he finally reached a whaling station on its north coast, whereupon he promptly borrowed a ship to mount a rescue mission. That rescue was frustrated by impassable ice, as were three further attempts. But in August he finally retrieved the 22 men still waiting on Elephant Island.

Behind this practicality lay persistence. This incredible derring-do would never have been possible had Shackleton and his crew not shared a common bond and strength of purpose. Everything that Shackleton achieved, he achieved with and because of those he had taken with him – 27 men chosen from more than 5,000 applicants. What did he look for? “The quality I look for most is optimism,” he said, “especially optimism in the face of reverses and apparent defeat. Optimism is true moral courage.”

Not many of us will have our mettle tested as Shackleton and his team did. But we all have our reckonings with life and death sooner or later, or other adversities that make us reappraise the world and question the future. It’s at such times that optimism can be hardest to secure, but also most valuable. Optimism, far from leading us to passively await our fates, can help us to actively explore our limitations – and transcend them.

“Once I knew only darkness and stillness. Now I know hope and joy,” wrote Helen Keller in her 1903 essay Optimism, 21 years after the childhood illness that had destroyed her sight and hearing. “My life was without past or future; death, the pessimist would say, ‘a consummation devoutly to be wished’. But a little word from the fingers of another fell into my hand that clutched at emptiness, and my heart leaped to the rapture of living.” The fingers were those of Anne Sullivan, Keller’s teacher; and the “little word” they were spelling out was “water”, which was simultaneously running on to Keller’s other hand. Sullivan had previously worked with Samuel Gridley Howe, the doctor who had pioneered the finger-writing technique two decades earlier with Laura Bridgman, another young deaf-blind girl. Keller’s mother had read Charles Dickens’s touching account of their success, and sought help for her own frustrated daughter, then aged seven.

In her essay, Keller suggests that Howe “found his way to Laura Bridgman’s soul because he began with the belief that he could reach it. English jurists had said that the deaf-blind were idiots in the eyes of the law. Behold what the optimist does. He converts a hard legal axiom; he looks behind the dull impassive clay and sees a human soul in bondage, and quietly, resolutely sets about its deliverance.”

Optimism, Keller suggests, “compels the world forward, and pessimism retards it”. Pessimism for a nation, as for an individual, “kills the instinct that urges men to struggle against poverty, ignorance and crime, and dries up all the fountains of joy in the world”. Optimism, by contrast, is “the faith that leads to achievement”. Without it, nothing can be made better. Her description of optimism’s power, effects and practice still resonate today.

For all this talk of the power of optimism, however, there’s a problem. Optimism is associated with unexpected victories, mountains moved and triumph over adversity; but also with unkept promises, unaffordable bets and unrealised dreams. Our conception of optimism is about positive expectations of the future and of course we can never prove that those expectations are well-founded before the fact. When things turn out well, we praise the optimism of our political, social and commercial leaders as inspirational; when things go badly, we disparage it as wishful thinking.

One way to make the case for optimism is to acknowledge that there are things we don’t know, that some of those unknowns are positive and that we have some ability to steer towards those positives. Optimism encourages us to seek them out. If, on the other hand, we have no expectation that our lot in life can be improved, we have no motivation to put in the thought and effort needed to improve it and those solutions go undiscovered. Failure becomes a self-fulfilling prophecy. Pessimism traps abound in human lives. Jobs you don’t expect to get, so you never apply; crushes whom you believe to be out of your league, so you never ask them out; games you expect to lose, so you never play. From this point of view, it’s not surprising that optimists turn out to be more successful than pessimists in almost every aspect of their lives. They tend to do better at school and at work; they have stronger relationships with family and friends; and they’re more resilient in the face of financial, mental or physical stress. It’s the stuff of cliche; 80% of success is just showing up. You miss every shot you don’t take.

When I was a kid, I sometimes went up to my dad’s offices during the holidays. One day I stumbled across a stack of New Scientist magazines. This was a novel and exciting concept: I liked science, and I liked writing, but was dimly aware that these were not widely regarded as compatible life skills. I’d be lying if I said NS was kid-friendly, but its pages were stuffed with discoveries and inventions; at the back there was a comic strip and a column by a mad scientist. This, I decided, was what I’d do when I grew up. I asked my dad who took care of a magazine like New Scientist. The editor, he said. So I decided that was the job I’d do.

In due course I went off to university to study physics, then I switched back to writing. By now it had dawned on me that being the editor of New Scientist was not a role you just waltzed into. In fact, getting any position there seemed like a stretch. But I lived in hope. So I wrote to the features editor and asked for a job, on the basis that I liked science and I liked writing. He wrote a kindly letter back, suggesting that I should get some experience first. I applied for the first job in journalism that I could get, which turned out to be for a boring financial magazine. But the late 90s were a good time to be an aspiring writer: I worked with other youngsters, I could pay my bills, and I got to see the world while doing nothing more arduous than cranking out words. I fell in love, got married – and then everything changed in 2005, when Kathryn died.

While I was picking myself up, I saw an ad for a temporary job at New Scientist. Having worked in finance for more than a decade, I couldn’t see how I was suited to the gig. But I had nothing left to lose, so I applied. And I did get the job, from the man I had written to. Working at New Scientist proved to be both just like I’d expected and absolutely nothing like it: but in any case, one thing led to another and I ended up, some 30 years after my first acquaintance with the magazine, serving as the editor-in-chief.

Does this testify to the value of optimism? Perhaps it does: a self-fulfilling prophecy enacted over decades. Or to put it another way, there were any number of points at which I could have succumbed to a pessimism trap. You miss every shot you don’t take. Or perhaps I could have got the job much sooner if I’d been more optimistic – and perhaps in another world I did; but I don’t have access to those possible worlds. Just this one.

In this one, I spent the best part of a decade living out my childhood ambition. I found a new direction, a new purpose, a new love and a new family. Is my life today perfect? Of course not: but I try to make sure it’s as good as it possibly can be. We can’t control the happenstances that determine the course of our lives. But we can control how we respond to them, and we can carry on looking for the bright side, no matter how dark the future appears. We can’t make a perfect world. But we owe it to ourselves, to those around us, to those we have lost and those who have yet to come, to make the best of it that we can. The best of all possible worlds.