How To Become A Software Engineer Expert—The Pyramid of Mastery

Previously I defined The Pyramid Of Mastery—a model that describes the aspects of what does being an Expert mean.

But a model without some specific examples of how it applies to the real world doesn’t really mean anything. Therefore, in this post, I will provide an example of what does it mean to master the Software Engineering domain. To narrow things down a bit, let’s focus on Software Engineering in Web Development.

A word of caution before we begin: a domain is immense, and even if we narrowed it down to Web Development, it still would take volumes to describe it entirely and in all details. Therefore, this article does not intend, nor does it claim to be an overarching description of all the knowledge and experiences one should possess to become a Software Engineering expert. It does, however, offer a good overview and a general feeling of what it takes to master the field. You can treat it as a movie review (but with spoilers)—it offers a glimpse of what it takes to go through the experience, but the actual journey is more vibrant and eventful.

Additionally, except for a few things at the beginning, there is no predefined point in time that indicates when learning about some concepts should happen, so the path described below is one of the millions of potential tracks.

A Brief Recap Of The Pyramid Of Mastery

If you haven’t read the original article on The Pyramid Of Mastery—How To Become An Expert In Your Field, I highly recommend you do.

Overall, it states that any domain consists of:

• Elements. These are the building blocks of the field that include concepts, ideas, and entities;
• Rules. These are the laws that govern a domain. This layer comprises both rules that define the interaction between various Elements as well as organizational and governing rules at the domain level;
• Tools. These are the instruments you use to perform the activities within a field;
• Frameworks. These are combinations of the above levels that allow for faster achievement of preset goals for which the Framework was created;

The Fundamental Elements

There are countless things an expert software engineer knows, and that knowledge accumulates throughout the years. But pretty much all software engineers start from the same place - learning the fundamental concepts of programming. And so will we.

If you are not familiar with the domain of software engineering, there is a standard first “program” a person writes to get started with a programming language, called “Hello, world!” Essentially, you have to write a program that outputs “Hello, world!” on the screen, which illustrates the basic syntax of a programming language.

Any beginner learns there are the following Elements (in no particular order):

• variables;
• values of different types;
• control flow statements;
• loops;
• functions;

You do that while learning some Rules in parallel:

• how variable assignment works;
• usage of arithmetic operators between values of different types;
• boolean operators' logic;
• etc.

Usually, you start your learning journey by choosing and using a specific programming language you will be working with and a text editor or an IDE (Integrated Development Environment). These are the main Tools you have to learn (the syntax of the programming language, the panels and features of the IDE, etc.) on top of acquiring knowledge about the essential Elements and Rules.

There is an interesting observation at this point (and it keeps popping out throughout the whole learning journey): a Tool is as useful as your understanding of the underlying levels this tool covers.

When you dive into programming and learn the essential elements and rules (as in the example above)—you only know how to write basic statements. The programming language you are using is incredibly powerful and capable of much more, but all those features are useless if you don’t know what they are and how to use them.

The First Learning Cycle

So, you learned the basics, and you start practicing your newly acquired skills by doing some exercises to obtain experience using your recently gained knowledge. As you will see below, this cycle is an iteration you will go over and over in your path toward mastering the Software Engineering domain.

And thus, you begin building your pyramid of understanding of the Software Engineering domain, one step at a time. It is quite narrow now, but the more you learn, the bigger and wider it gets.

Once you gained a good understanding of the basics, you go back to learning more elements.

Going One Step Further

You continue learning more Elements. A widespread programming paradigm today is Object Oriented Programming (OOP), based on the concept of “objects” that can contain fields of data (called attributes) and functions (called methods) that can modify that data. And so you start learning about more Elements: classes, object instances, methods, properties, the trio “foo, bar, baz,” constructors, etc. All that along with some Rules: the mechanism of inheritance (single, multiple, multilevel), object composition, encapsulation, visibility modifiers, polymorphism, mutability vs. immutability, etc.

You might also find out this early that OOP is a type of Imperative Programming, which contrasts with the Declarative Programming paradigms, which opens a whole PP Pandora Box (Programming Paradigms Pandora Box :) ) of its own. You take the wise decision to leave it as is for now and get back to it later.

Along learning about OOP, you learn about some data structures: hash tables, sets, arrays, stacks, queues, trees, tries, etc. and the rules by which they abide.

Now you can build programs that are a bit more complex, and you put in practice all the newly acquired knowledge until you get a better understanding of those Elements and Rules. Your pyramid gets wider as you study more things.

Taking A Huge Leap Forward Just To Realize It Is Only A Little Step

You know the basics, and you feel confident you can do anything now. If you could understand all that boring stuff about classes and inheritance and encapsulation—what can be harder than that?

You decide to do something more exciting—to tackle a web Framework and build something real—your own site. You choose one based on the language you have learned so far, install it, follow the tutorial, and in under an hour, you have your first web site up and running on your computer! Easy-peasy! How cool is that? I bet at this point you feel almighty. However, that is a deceiving feeling that soon will mutate into a lingering and frustrating period of learning about how the Internet works.

Brace yourself, as you will quickly begin to question your decision to master web development. The moment you need to make a change that is not in the tutorial, you will realize there is a lot of magic you do not understand that happens behind the curtains. And that what you do not understand you cannot tame. Therefore, it’s time to learn more Elements and Rules!

The Internet Basics

There is an avalanche of information you need to know about how the Internet works, but not all of it is necessary to build a simple website.

The “What You See” Side

So you initiate yourself with how information gets displayed and learn about the following Tools: HTML and CSS that consist of different tags, attributes, styles, and the Rules they follow. By playing with HTML and CSS, you gradually gain experience, and this knowledge alone offers you some degree of flexibility by giving you the freedom to build whatever you want. There is another pitfall you are going to learn soon—unfortunately, sometimes the way you see the page varies from how others see it on their screens because different browsers render the pages differently. You do some research on this issue and find out there are entire Frameworks that deal with those cross-browser compatibility issues. You do more research and find out a list similar to this:

• Bootstrap;
• Materialize;
• Foundation;
• Semantic UI;
• another 5-10 frameworks;

Because of the inflow of information about what those frameworks can do, and because you decided to become a Full Stack Engineer, you somehow pick one and learn just enough to accomplish what you want. I’d suggest you bookmark that framework’s documentation page because you will get back to it regularly.

Through trial and error, you style the website based on how you have envisioned it from the start (actually, you style it as you go, changing the design in the process multiple times). You would like to add some interactivity and animations to the site, but it looks like you need to learn JavaScript and another framework to do that. You decide to take a break from the frontend stuff and focus on the backend.

The “What You Do Not See” Side

So you have a more or less general understanding of the frontend now, and you continue your path toward mastery by starting a more complex site (e.g., a blog) and learning new Elements:

• request/response communication;
• protocols;
• Universal Resource Locator (URL);
• Roles and Permissions;
• User Authentication Methods;

And the Rules:

• the URL syntax;
• the synchronous vs. asynchronous (sync vs. async) calls;
• the code execution flow in request/response communication;
• etc.

You might learn that the framework you are working in is a Model-View-Controller (MVC) type of framework, so you also study some code organization rules and the DRY (Don’t Repeat Yourself) principle.

Although there is much more to learn about the Internet, you have to step back from it for a while, as you cannot proceed further in developing your blog without learning how to store the information you will write on your site.

The Storage

The information that displays when you access a web site can live either inside a database (text, numbers, dates) or on disk (images, music, files).

You dive into familiarizing yourself with the databases by learning about the following Elements:

• tables, rows, columns;
• relations;
• queries;
• indices;
• transactions

Additionally, you find out about Structured Query Language (SQL, which is a type of Declarative Programming Language mentioned earlier), and NoSQL (Not Only SQL, but you better ignore that for now) and a whole flood of new concepts: clauses, expressions, predicates, statements, stored procedures, etc.

And Rules:

• uniqueness constraints;
• SQL syntax;
• creating, altering, dropping tables;
• normalization;
• etc.

The framework you are working with will probably offer an Object-Relational Mapper (ORM) for some databases (DBs), which will simplify the manipulations with data in your project (at the expense of abstracting away the implementation details).

So you choose a database and continue to work on your project. You go through a lot of learning and frustration until you have a working version that you want to deploy on the Internet. You learn about domains, Domain Name System (DNS), IPs, web servers, Transport Layer Security (TLS), Secure Shells (SSH), and many other things before you manage to put your web site online. You are happy and proud of yourself, but this is only the beginning.

There Is A Lot Happening In Parallel

Learning is not a linear process, and there are many things you are going to learn sporadically, along with all the concepts I have described previously.

Some of the Elements you’ll probably learn:

• algorithms;
• notions of space and time complexity of algorithms;
• unicode;
• encodings;
• tests (unit, integration, e2e);
• graphs;
• events;
• Document Object Model (DOM);
• etc.

Some Rules to complement the list:

• bitwise operations;
• how to sort arrays;
• how to traverse a tree/graph;
• how to test code;
• encoding/decoding characters;
• etc.

You keep learning about the Tools you use:

• improving your knowledge of the standard library of the programming language of your choice;
• learning about the Relational Database Management System (RDBMS) of your choice;
• web servers;
• start learning JavaScript for frontend development;
• understanding the power of pen and paper (or marker and whiteboard);
• etc.

And, of course, the Frameworks:

• improving the knowledge of the web framework of your choice;
• learning to use some testing frameworks;
• using some third-party libraries or modules within your project;
• getting a better understanding of some cloud service of your choice;
• etc.

A Few Words On Software Engineering Titles

Up until now, your pyramid looks like the following:

Although there is no clear boundary as to what makes a junior, a mid-level, a senior, and all the rest software engineering statuses, the pyramid helps define, to some extent, the level of a developer.

For getting started in any domain, you need to have a good arsenal of Elements from a field. Therefore, once you get a good understanding of enough Elements in Software Engineering to be able to do some work, you are a well-established junior developer. You might still need some guidance from a more senior developer, as you lack some experience and understanding of some more Rules, but you are proficient with the basics and can already bring value to a team.

Widening The Learning Cycles

Interestingly, as you continue to improve your knowledge and study new things, the frequency at which you learn and switch between different layers gradually widens once you get a good grasp of all the essential knowledge to work on projects.

You still learn new Elements, Rules, Tools, and Frameworks, but you also progressively spend more time applying the knowledge and experience you already have. That’s because you are competent enough to get real work done and so you use your expertise to solve real-world problems.

Upping Your Level By Deepening Your Knowledge

You gradually become more knowledgeable and experienced as you continue to work on different projects and learn new things. But throughout your path so far, you might have noticed that there are concepts you encountered for which you have barely scratched the surface. Therefore, you will have to go back and tackle those in more detail, along with learning new stuff.

Speaking about new programming concepts:

• memory allocation (pointers, stack and heap memory, dangling references, dereferencing, garbage collectors, borrowing, etc.);
• concurrency (shared memory, message passing);
• parallelism;
• functional programming;
• recursion;
• higher-order functions;
• decorators;
• prototypal-based inheritance;
• caching;
• etc.

You will also improve your understanding of how the Internet and web servers function:

• Protocol (Network) Stack (with its seven layers);
• more application layer protocols (FTP, LDAP, POP, SMTP);
• transport layer protocols (TCP, UDP);
• IPv6;
• sessions;
• cookies;
• middleware;
• encryption;
• Application Programming Interfaces (APIs);
• etc.

And also learn more storage concepts:

• Materialized Views;
• vertical vs. horizontal scaling;
• Atomicity, Consistency, Isolation, Durability (ACID);
• and get a glimpse into the NoSQL DB landscape, with its own Elements and Rules:
• Key-Value stores;
• Document Databases;
• Columnar Databases;
• Graph Databases;
• etc.

You also begin to learn broader Rules that affect not only specific Elements, but whole projects:

• code organization principles;
• design patterns;
• software architecture styles:
• monolith;
• Service Oriented Architecture (SOA);
• microservices;
• event-driven;
• Representational State Transfer (REST);
• etc.

You have enlarged your horizons, and now you know this much:

A Note On Software Engineering Domain Utility

Each domain exists because it solves a specific set of problems. Web Development offers people and companies a way to reach out to millions of people, stay connected, do business online, and provide information. Therefore, a software engineer’s ultimate goal is to solve a problem that a person/business has.

When you just start working as a software engineer, the reach of your activities is narrow: separate functions, methods, small files, tests, etc. But gradually, as you become better at what you do, the scope of your work broadens as you begin to handle entire services, APIs, modules. Once you have successfully done that, you begin to define complete architectures: initially for small projects, and then for massive, scalable, distributed systems.

There are many discussions about what makes a Junior, Mid-level, and Senior Software Engineer. Many rely on the experience in terms of the number of years a person has worked in the domain. That is a vanity metric. Although more years of experience might correlate with the professionalism level of developers, it does not describe the level of expertise of a person. The things you know and the scope of the problems you can solve do describe it much better.

And that is a natural way of things—larger projects consist of a more substantial amount of Elements and Rules, and to be able to tackle those, you need to understand all their constituent parts.

Advancing Further

So you are quite good now, but there is always space for improvement. There are still more concepts out there that you either just barely heard about, or have to discover yet.

You will dive into the land of distributed systems, with a whole range of new (or just superficially known) concepts:

• Consistency, Availability Partition tolerance (CAP theorem);
• Basically Available, Soft state, Eventual consistency (BASE systems);
• load-balancing;
• Content Delivery Networks (CDNs);
• replication;
• circuit breakers;
• sharding;
• table partitioning;
• latency and throughput;
• distributed tracking, tracing and measuring;
• distributed security;
• searching;
• failover;
• autoscaling;
• etc.

You will also improve your knowledge about the programming languages (at this point you have worked with several of them already), how they work internally, when they shine the most and when it is better to use another tool to solve the problem.

And you will also learn other concepts from the domain (if you haven’t done that, already):

• serialization
• Remote Procedure Calls (RPCs)
• Web Sockets
• message queues;
• containers;
• virtualization;
• Software Development Cycle;
• futures and promises;
• etc.

The list can go on and on, but let’s stop here.

Attaining Mastery

It was a long and exciting journey that led you to a deeper understanding of the Software Engineering domain:

But, as you can see in the image, it doesn’t end here. Mastery is an ongoing process that requires a thorough approach toward your work and extensive practice. There are many more concepts that I have left out, and I haven’t discussed almost anything about the frontend aspect of Web Development, which has plenty of additional Elements, Rules, Tools, and Frameworks in itself! If you’d like to do that yourself, I’d love to take a look at the final result!

And a final word to those who either plan to become software engineers or are at the beginning of their careers as developers. It might look hard and different than what you have expected. But bear in mind that the path I have described here takes years to go through. Besides that, becoming an expert requires a lot of effort in any domain. But with enough perseverance and determination, you will be able to do it!

If you liked this article, consider subscribing below and following me on twitter (@iuliangulea).

---