If you’ve seen Idiocracy (Side note: I am not recommending the movie; I’m using a phrase from it we’ve all heard. The movie itself was a comedy not at all meant to be family-friendly when it was made, and changes in social mores since its 2006 release make it far less acceptable today), you are familiar with the title phrase I’ve used here. That’s what is repeated almost religiously to justify why they buy/use/drink Brawndo, an energy drink.
One of the scenes involves people telling the star that plants need Brawndo—“It’s got electrolytes … That’s what plants crave!” This is repeated over and over until he asks, “What are electrolytes?” After some stuttering and failures to answer, someone says, “It’s what they make Brawndo out of.” Someone else inevitably tacks on, “It’s what plants crave.”
Now, substitute “AI” for “Electrolytes”. That’s where we seem to be. Ask a vendor what algorithms they’re using, where their training data comes from, the difference in results between the test reservations and the real world. Most will end up saying a lot, not answering your questions and ending with, “It’s what staff crave!” or some such.
A few companies will lay it all out there, assuming you can parse through the information and are familiar with the AI training algorithms they are using or the algorithms they’re using to classify inputs against the dataset. And these companies are grand because they (a) know what they’re actually doing beyond “It’s got AI/ML, it’s what staff crave,” and (b) Expect that you are knowledgeable about the data you’ve asked for or are capable of looking it up in this age of wonder that has boundless information at your fingertips.
Don’t let your vendors get away with just repeating “It’s got electrolytes” over and over. AI and ML are a massive help in some areas, but they’re not yet a panacea and success rates vary wildly whether we’re talking problem domains, datasets, algorithms, training style, etc. Make them tell you the details. Where the data comes from and how big that dataset is. How much was reserved for testing and whether it was reserved randomly. Ask them what they hope AI will achieve, then ask them how that correlates to human interaction. Most AI today is still evaluating or replicating human interactions with something—most AI in IT particularly so. What algorithms were used to validate? And finally, make sure to ask for real-world results. I know people in data science and one has confidently said to me, “My models are 95% accurate.” When I finally dug into that claim (because my experience is that real-world use of AI/ML is almost never that accurate) it turned out that claim was true … for the reserved test data. So, the same dataset used to train got the same results on the reserved subset. That’s a good thing, but it is no indication of how said algorithm will do in the wild with unfiltered data. The big problem really is the switch from test data to real-world data.
In fact, in IT, we are a little better off than most spaces wrestling with AI/ML, because very often there is no difference between test and training data in IT. A network security vendor will test against actual traffic—because they have access to massive volumes of it and it shortens the iterations to use what’s at hand and what the system is being trained for.
And keep rocking it. The current iteration of IT is the opposite of the dark comedy future depicted in Idiocracy. You all are putting up systems and keeping them running at increasingly fast cadences; sometimes remotely because of COVID-19 or other work-related issues. Don’t stop turning out success for your coworkers and customers and take a moment to force vendors to explain what they’re doing with AI. If you’re a generalist like me, ask over email, though, so you can go look up the answers and make sure you understand the details.
