Understanding the Top Data Scientists: Challenges, Skills, and Uncommon Wisdom

In the world of data science, there are distinct types of professionals who tackle problems in unique ways. Manythough not allcome from a programming background and delve into the latest machine learning libraries. However, the best data scientists possess a level of wisdom that transcends their technical prowess.

Machine Learning: Weapons of Mass Distraction

It is often the case that all data scientists face challenges with their models, from the brightest to the less talented. The most brilliant of them may not understand why their models misbehave, and the least talented may not even be aware they do. A data scientist labels themselves as such more often derives from a programming background rather than deep domain expertise.

As a result, these individuals might focus on the latest machine learning libraries and a few tricks from recent Kaggle competitions. Their projects often achieve excellent accuracy at the start, earning them praise but often crumbling within a few months as they move on to the next opportunity. This pattern may or may not be fictional but is undoubtedly familiar to many.

It is important to recognize that the best data scientists know when to not use machine learning. This applies to all aspects of the process, from using 'just counting' to heuristics, as well as Bayesian statistics or operations research.

Data Science: Not Magic, Just Good Data and Smart Questions

Data science isn't always about big data; it's about clean data. Most data is unstructured, and the systems reveal our existing biases. Consequently, conclusions are tentative, insights are fleeting, and making silicon sweat does not guarantee results. The quality of the data and the questions being asked are what truly make data science effective.

Data science is not a magic wand to fix all problems. It is only as good as the data you have and the questions you want to answer. As a data scientist, you must be aware of your human capital and their complementary skills. Not all humans are created equal; some are extremely smart and adaptable, while others are stubborn and unteachable. The best data scientists constantly assess their human resources and invest in those who can deliver value.

The Insider's Perspective: Consistent Revenue as the Ultimate Metric

The best data science teams understand that consistently high revenue is their ultimate metric. If a data scientist or a team doesn't deliver 3x the revenue they are paid, they will eventually be let go. Even in the consulting world, there is pressure to generate revenue. If you don't produce results, even with no boss directly supervising, you won't stay in the game.

Data science is a high-stakes field, but the best professionals understand that it is not just about the technology. It's about the data, the questions, the human capital, and the ability to deliver consistent revenue. By focusing on these elements, data scientists can create a sustainable, valuable career path in a field that is constantly evolving.