AI calculates smart solution to manage digital imagery

It’s been a picture perfect year for Smartvid.io, a 2015 Cambridge, MA-based start up that incorporates machine learning, a form of artificial intelligence, to quickly and easily catalog, retrieve, and analyze imagery captured by drones, wearables, tablets, and other devices on construction sites

The enterprise, whose machine learning facilitates automatic and voice recognition to classify—or, “SmartTag”—imagery, joined project-management software giant Procore Technologies‘ App Marketplace in October. A month later, mega-contractor Skanska USA forged a collaboration with Smartvid.io to support use of its platform at select project sites under the auspices of the Skanska Innovation Grant Program.

The initiatives come at a time when the use of digital visuals to document work, perform inspections, and complete site assessments is proliferating at unparalleled rates due to the rapid growth of technologies generating the imagery, according to SmartVid.io Founder and CEO Josh Kanner, who will serve as a panelist at BuiltWorlds Summit 2017 in Chicago this week. The problem is, “the process by which engineers, contractors, and other construction professionals manage resulting imagery hasn’t changed much in the past 10 years,” Kanner says. “You can only search by file name or by who took it and when.” Facilitating file searches frequently translates into the time-consuming process of manually renaming images to facilitate their retrieval. Short of that, searches rely on remembering the precise date imagery was recorded. Further, photos often are stored in files on shared directories that require manual management.

The Smartvid.io platform, by comparison, expedites image retrieval. The platform owes much to the likes of Google and Apple, enterprises that implemented machine-learning initiatives—processes by which the technology learns from its mistakes to continually improve its functionality—to ensure search engines in the “consumer world” quickly and accurately retrieved visual information, according to Kanner. The concept of a similar engine dedicated to construction began to ferment when Kanner asked himself, “If you can search for an image of a cat, why not achieve the same speed and accuracy for a crack in a structure?”

The question prompted development of a SmartTag engine that leverages machine learning to tag visual content on the basis of audio narration, including key words such as “ductwork” or “drywall” and precise locations to describe corresponding imagery collected during video- or photo-based documentation. Alternatively, the software, nicknamed VINNIE, short for Very Intelligent Neural Network for Insight and Evaluation, can recognize and tag certain components on its own, based on a tag-based project dictionary, Kanner says. Additionally, all imagery is automatically dated, allowing users to chart progress by comparing conditions in a given space over periods of time. All users need do is upload the imagery—up to several videos at time—via Smartvid.io’s app or a mobile web version of the software. On the back end, VINNIE works its magic, matching what is seen or said and classifying information accordingly, dramatically reducing processing time.

Information retrieval is simple. One method involves creating links and locating them in a BIM 3-D model. Once in place, users need only click on the given space to access related visual information. Once imagery is received, a corresponding field denoting voice and image recognition appears to the left, and a field for commentary appears to the right. Users also can mark up the video or photo.

Stakeholders can easily share files, including the results of multitagged searches. Depending on circumstance, recipients may only view the desired duration rather than the entire file.

“Procore’s Construction Operating System is about openness,” said Laura Paciano, senior product marketing manager with Procore. “By integrating with innovative applications like Smartvid.io, we’re providing all Procore users with a choice that allows them to work with the tools that best fit their construction business process.”

The platform remains a work in progress, with enhancements including the development of a safety-recognition program for job sites. “The program currently can identify people on a site and whether they are wearing hard hats and safety vests,” Kanner says. In 2016, Smartvid.io teamed with construction weekly ENR magazine to put those capabilities to the test in an evaluation of safety conditions relating to the publication’s annual Year in Construction Photos Contest. Among the criteria for winning entries is safety, with experts on hand to ensure images don’t depict hazardous conditions. Scanning for safety vests and hard hats, or lack thereof, VINNIE sorted through some 1,080 submissions in less than 10 minutes, compared to the 4.5 hours required by the human team. Additionally, VINNIE correctly identified 446 images with people, while the team detected 414. Finally, VINNIE identified 32 instances of field workers missing hard hats and 106 images with workers missing safety-colored vests.

Plans are to expand Smartvid.ios safety capabilities to automatically identify standing water, which is conducive to falls and other potential hazards, Kanner says.

Another initiative under way seeks to evaluate productivity, with variables including personnel and equipment and the manner in which they are deployed. A third initiative involves quality, particularly the identification of defects. Smartvid.io already has worked with engineer Arup to detect defects in tunnel walls during inspections.

In a short period, SmartVid.io has garnered considerable attention among industry members, with other clients such as Suffolk Construction and, in addition to Procore, other partners such as Autodesk and Egynyte.

In 2015, just months after its launch, Smartvid.io raised an initial round of $3.4 million in financing led by CommonAngel Ventures, with participation from Launchpad Venture Group and other investors. “We’re always excited when enterprises show interest in assisting in what we do, “ Kassner says. “And there’s still so much to do.”