Emerging Tech Is Nothing Without Methodology

Or: a Hundred Ways to Solve a Complex Problem

For years I have introduced myself as an analytic methodologist. This aligns with both my formal academic training and my chosen trade. The assertion has been met with confusion, curiosity, and at times disapproval. For many, a methodologist is synonymous with a generalist, which in the world of technology implementation and all eyes on AI, no one much likes to be.

Traditionally, methodologists are those who study research methods, both qualitative and quantitative. The word “research” etymologically speaking means ‘to go about seeking:’ a “creative and systematic work undertaken to increase the stock of knowledge.” (OECD Frascati Manual 2015)

A creative and systematic work…

Practitioner methodologists, independent of their association with research methods, are encyclopedias of ways to approach complex problems. A method is a way of doing something; an approach. I contend that at the nexus of sound science and quality solutioning in any industry is methodology. The remainder of this article advocates for methodology as a discipline.

On Methodology

When designing technical or analytical solutions, we are often working backwards from what we want to achieve. Good science says to put the problem first and then select relevant methods to reach a viable solution. We are to then implement those methods using the corresponding technologies populated with the data that the method(s) require. In other words, data feeds the technologies, technologies implement the methods, and the combination of methods solves the problem.

For example, if the problem we are trying to solve is the spread of COVID-19, we may pursue the distillation of a contact tracing solution as outlined in the image below. The candidate solution of contact tracing could involve two methods: 1) social network analysis of contact, and 2) mathematical modeling of contagion (e.g., SEIR model). The technical implementation of these methods will involve selected technologies or software products, and the relevant datasets. The work of designing the conceptual solution is that of methodologists and data scientists. The work of designing the technical architecture is that of solution architects and engineers.

The strength of analytic methodology is in identifying multiple relevant methods to solve a problem and understanding what technical components are required to bring those methods to fruition. It requires both creativity and a systematic process to comprehend multiple approaches, test them quickly, and promote one towards the ultimate solution.

In a research project, this process would be the work of several years and multiple academic publications. In a technical project, this should be the work of a few weeks. It requires a scientific mindset and an agile aptitude for creativity and experimentation.

Methodology and Data Science

So what is the relationship between analytic methodology and data science, or “AI/ML?” We see machine learning (ML) and artificial intelligence (AI) getting a lot of attention these days. From a methodology perspective, we are able to put AI (as a domain of science) and ML (as a collection of methods), in their places alongside other technical methods. Even the coveted generative AI is simply an incremental development of unsupervised learning, albeit quite an innovative one.

As a methodologist myself, I always found it odd that ML got so much attention while other methods remained in the shadows of industries (like agent-based modeling…). The Department of Defense found it special enough to create an entirely new organization: the Joint Artificial Intelligence Center (JAIC), now the Chief Digital and Artificial Intelligence Office (CDAO). There are congressionally-appointed funding streams for the application of ML algorithms and generative AI.

I don’t know of another method that has its own congressionally-appointed funding stream. So why is AI so special?

The methodologists’ answer: it’s not. The context-appropriate answer: it’s complex.

ML algorithms handle data volume in a way that humans cannot. In return, they require a lot of computational power. And really good data. Ultimately, ML algorithms are computational implementations of complex math. This means that the results of complex math are now in the hands of analyst users. That, I would argue, is a little special.

ML algorithms can also evolve beyond their intended training or purpose, which is something other methods cannot do. This is the “learning” in machine learning, and the “generative” in generative AI. But the most arresting feature we now see in this class of methods is in language generation. Regardless of the actual capability or comprehension of a large language model (LLM), it speaks our language. And when something speaks to you in your native tongue, something about the experience engenders trust. #anthropomorphism No other method speaks back to the methodologist in plain English.

While these things do make AI a unique domain of science that contains a unique suite of analytic methods, ML algorithms are still methods at the end of the day, and they are not suited to every problem. There is still a need for the methodology mindset in applying these methods where they are fit-for-purpose and applying other methods where they are not.

On Methods

There are many, many methods from which we methodologists distill creative solutions across industries. I’ve written previously about graph analytics and entity resolution, the first of which is an analytic method, the second of which is more of a data engineering method. There are traditional methods (e.g., simulation, cluster analysis, time series analysis, sentiment analysis). Then, of course, there is machine learning (supervised, unsupervised, and reinforcement learning), and a suite of statistical forecasting methods. There are cognitive thinking strategies (e.g., perspective taking, role playing, analysis of competing hypotheses, multi-criteria decision matrices) and more practitioner-focused capabilities (e.g., geospatial modeling, pattern-of-life analysis, advanced data visualization techniques).

Though by no means exhaustive, these approaches are applied differently from one industry to the next. Ultimately, they are Lego pieces intended for a methodologist to assemble into the grandeur of a solution to whatever challenge the industry or enterprise is facing.

So how do we bring sound science and high-fidelity methodology to quick-turn technical solutioning when there is an imminent deadline?

Too many times we launch data-first efforts. ‘We’ve got these two datasets; what can we learn from them?’ While that is a perfectly valid question to ask of the data within an organization, it isn’t necessarily the best setup for scientifically-grounded inquiry and solutioning.

For accelerated research, rapid prototyping, and quality solutioning, your organization requires the methodology mindset to orient around the problem and begin with the first principles of a solution. Without methodology, inundated with emerging technology, we are all just going faster and further away from the point.

Emerging Tech Is Nothing Without Methodology was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.