What I Learned as a Software Engineer

Lessons taken from books, others’ stories and my personal experiences.

Sunnyvale, May 07, 2022

This post is a collection of lessons I learned over the past few years since I graduated from school and started my career as a software engineer. Most of them are philosophical and generally applicable, although some are characteristic of my work in tech. They are mostly subjective and only reflect my beliefs at this moment, so readers should take them with a grain of salt.

Understand the problem before trying to solve it.

No matter how pressing a task is or how easy a problem seems, always make sure you clarify all the requirements, collect enough data and make thorough analysis before you work on your solutions. Oftentimes we see newbie programmers diligently engaged in voodoo programming, experienced engineers fervently pitching their theories before any investigation, and talented people working on sound solutions to the wrong problems. These people may have a speculative motive to reach a goal through some shortcut, or they may have more passion for exhibiting their calibre than knowing what’s actually going on. As a result, sometimes they produce crack cures that fail in the long run, or, more often than not, just run into fiascos immediately.

In my opinion, gaining an understanding and making some achievement are two sides of the same coin, just as state estimation and optimal control are dual problems in certain settings. The more we know, the greater power we have to achieve what we want, and in turn we are able to research more things. So if you want to be a brilliant problem solver, start with being an inquisitive learner and critical thinker.

There is a single version of truth, but could be multiple valid perspectives.

Everyone’s knowledge about the objective world is just an approximate model. As human beings, we have to save mental energy by building simplified concepts and narratives to interpret what we observe. Even established scientific theories are based on limited samples in the experiments and thus have a bias. Inevitably, every theory has its limitation, but on the other hand, each tends to have its own virtue. For example, Newtonian mechanics is still widely used in practice due to its simplicity, even though we know it does not universally hold true. Likewise, in everyday life, we may easily overlook other opinions and perspectives when we already have a theory in mind. Sometimes our understanding might be sound enough, but listening to different voices is always beneficial since it helps us to spot our blindspots and step out of the local optimum.

In Pursuit of Rationality

Stick to the problem rather than your solution.

As an engineer, we are both scientists and artists. When we look into a problem, we are doing an evidence-based scientific research. And when we work on a solution, we are creating some art that balances costs and benefits, pros and cons. One mistake we might make is obstinately holding on to the solution, the design and the plan we have, ignoring new data, new circumstances and new requirements. Instead, we should adapt to the changing situation, since we, as problem solvers, are flexible, while the problem itself isn’t. I had quite a few experiences in the past where I had to dramatically change my original proposal half way through my projects in order to overcome some unforeseen hindrances and to lower the risks of late delivery. They were undesirable at first thought, but turned out to be wise decisions in retrospect.

Embrace Changes with Fast Replanning

Trade-offs are ubiquitous in software engineering.

Nature does not allow us to obtain too many good things at the same time. Just as with every other decision we make in life, software design decisions are full of trade-offs. In real-time computation, we trade accuracy/optimality for latency and memory. In storage engines, we trade write performance for read performance. In distributed systems, we trade consistency for availability. Examples can go on and on. Thus in everyday software engineering, no choice is absolutely right or wrong. Every tech decision has its pros and cons and deserves some debate.

In terms of project planning, there is also a trade-off between long-term development and short-term achievement. Only focusing on long-term development, we may find ourselves trapped into perfectionism and fall short of timely deliveries, thereby losing trust from our customers and jeopardizing our business values. Also, failing to deploy something early means missing the opportunity to obtain early feedbacks for further improvement. On the other hand, scurrying around for short-term performance will lead us to inelegant solutions, additional complexity and excessive technical debts. Therefore, a project leader needs to strike a balance between the two and get priorities straight based on the situation.

Software is more like gardening than construction.

I got this idea from the book The Pragmatic Programmer. We tend to think of software development as building construction, but it is actually more organic and flexible than that. Unlike buildings, software is up to change at any point and is seldom a faithful materialization of the original design.

In software engineering, we emphasize Separation of Concerns, by which we decompose a large system into separate components and consider one piece at a time. That not only simplifies the construction, interpretation and maintenance of the system, but also enables each granular component to evolve independently. Unlike traditional engineering efforts, software engineering attaches great importance to iterations. We don’t expect to build and deliver something once for all. Rather, we continuously launch better versions of a product as time progresses, and gradually approach an ideal product just as a numerical optimizer approaches an optimum. For one thing, software development takes time and we wish to deliver viable services to the market as early as possible. Moreover, real-world requirements are typically vague and capricious, so software designers typically don’t have enough information for making all the decisions in the early stage. In that sense, the evolvability of software is both a choice and a must.

The flexibility of software has lent great power to this industry. But on the flip side, its ever-changing nature also leads to unmanageable complexity. Take the robotic software I’ve been working on for example. Over the years, we have been handling countless unforeseen real-world problems. We tweak existing code and add new features/patches over and over, so much so that nowadays few people can clearly reason about how the entire pipeline would work in a certain scenario let alone knowing what needs to be updated if we want to change the behavior of the robot in some way.

As time goes by, the cost of making a new change increases dramatically, since each feature we are adding not only needs to work for the various external environments where our robotic system navigates, but also needs to be compatible with the assumptions and quirks of all the existing components. Changing the system in one aspect may introduce multiple ripple effects in other aspects, which in theory should have been orthogonal to the aspect being changed. And such cascaded changes are further amplified by various simulation scenarios we are testing (some are not realistic) and by different hardware platforms we are operating, causing a huge amount of alert noise that bombards our on-call engineers.

Such challenges are just for ordinary incremental improvements we make every day, let alone the occasional architectural changes whereby we wish to revert some pivotal tech decisions we made in the past. One lesson I take from this is that people in the AI industry should not only pay attention to the essential complexity that is unique to this field, but also take heed of the accidental complexity that is prevalent in all software engineering domains.

Plan ahead, but embrace unprepared situations.

Just as many other things in life, it takes deliberate planning and preparations to succeed in engineering projects. In order to move forward smoothly, we always need to look ahead and take upfront actions accordingly.

Nevertheless, don’t feel bad if things don’t go as expected. Life is such a chaotic system that many things may elude our prediction. A failure or mistake may turn out to be a blessing in disguise. In my personal experiences, the most triumphant victories typically did not take place when everything went well, but when things were tough and I somehow managed to stay resilient. My interpretation is that difficult situations trigger more alertness in our nervous systems, thereby boosting our performance. Meanwhile, when stakes are high, we become less risk-averse than when we are in comfort, so we take more initiative, grasp more opportunities and learn new things more voraciously. So just as Winston Churchill said,

Never let a good crisis go to waste.

Seek to understand people, not to judge them.

As software engineers, we don’t just work with code with machines, but also work with people. If you want to get better at predicting and influencing people, you need to take one step further than just judging them as right or wrong. Human beings have the attribution bias, meaning we have a tendency to attribute others’ behaviors to their personal dispositions, although we would attribute the same behavior of our own to objective situations. This bias comes with many downsides. First, we would be less empathetic than we could be, as we tend to interpret others’ mishaps as the result of their own mistakes. Second, we would fail to take lessons from others’ experiences and miss the educational values. Last but not least, we would develop a biased mental model for predicting people’s behaviors in the long run, leading to inadvisable decisions. So instead of making a judgement, stand in others’ shoes and reason about their behaviors in their situations.

Never try to influence others by showing them they are wrong.

If you want to persuade someone by winning a vigorous debate, or change their minds by pointing out their mistakes, please think twice. Human beings, as self-serving animals, don’t work in that way. We are such dumb creatures that we selectively ignore our own faults, find evidence to support our standpoints, and demonize those who make us feel bad about ourselves. Of course, that doesn’t mean everyone is insane and that striving for open-mindedness is a wild goose chase. My point is that such psychological bias is pervasive among ordinary people and should be taken into account for effective communications. To better communicate, start a conversation by showing consent and appreciation, and then insinuate your idea by letting people feel it belongs to their own thoughts. For more soft skill techniques, I recommend the classic book How to Win Friends and Influence People.

Leadership is more than power and mandates.

First of all, I’ve never been a leader or manager myself and never learned about leadership before. But as an individual contributor, I’ve observed both good and bad examples of leadership in different organizations, so I’d like to share my two cents here.

Leadership is far more than authority. It is more about people’s trust in you that you could help the team accomplish more and make everyone better off. Power is a tool to help achieve that goal but should never be overused, for doing so could overdraw your authoritativeness just as overprinting money will inflate a currency.

Good leaders lead by example. They are excellent problem solvers and hardworking employees themselves, and are always at the forefront of the crucial battles that their organizations are going through. They spend enough time with their teams, giving them guidance and support, and sometimes shield them from the pressure of upper management. They serve for the team just as everyone else, except they play a far more important role.

Good communication is an integral part of a successful leadership. That is especially important when rolling out a new policy or pushing forward a reform. A good leader should drive home to people the motivation and importance of the new policies, and also be honest about the prices or adverse effects entailed. They should allow open discussions about the matter and give everyone a chance to voice their opinions. They care about the feelings of the most noncompliant people, because they know discontent is contagious and that a small contention could be a broken window that cracks the morale of the whole team. On the other hand, when leaders treat even the toughest persons with patience, honesty and respect, their credits in the majority of the team begin to grow. In that sense, such a contention is so good a crisis that no leader affords to waste.

Thanks for reading this! If you enjoy it, please give me a clap and stay tuned to my future posts.