Nanjing's LightTok chip produces AI tokens inside the sensor pixel

The energy claim turns on where the power goes before a model sees anything. A study widely cited by other academics found that the analog-to-digital converter alone accounts for 66% of an image sensor's energy consumption on average ^[6]. Take the converter out completely and 34% of that budget remains, an improvement of about 2.9 times ^[18]. So the reported tenfold gain at converting light into tokens ^[8] cannot be the converter by itself. It also has to include the steps after digitisation that LightTok removes: the temporary buffering and the hop to a separate processor where the image is tiled ^[4]. Pushing that processing into the cloud instead raises total consumption ^[7].

The array is built from single-layer molybdenum disulfide floating-gate phototransistors, each of which senses light, remembers what it sensed, and factors that into a calculation ^[9]. Molybdenum disulfide is a 2D material that responds well to light and can be grown in sheets one atom thick ^[10]. The phototransistor converts photons into an electrical current, and the floating gate is an isolated component inside it that traps and holds charge instead of losing it once the light goes away ^[10].

"The chip physically eliminates data movement, which is the main source of energy waste," Liang Shi-Jun, a physics professor at Nanjing University, told the Chinese state news agency Xinhua ^[11]. "Light comes in, tokens come out," Liang said, which is where the name comes from ^[12]. Miao Feng, director of Nanjing University's Institute of Brain-Inspired Intelligence, said in a statement that the design idea "was to move token generation onto the sensor itself, allowing the chip to directly produce tokens that AI models can process once light reaches the sensor" ^[3].

The resolution is the weak spot in the claim. The array tops out at 32 by 32 photosensitive pixels, far short of a current smartphone camera, let alone a drone ^[13]. That is 1,024 pixels in total ^[17]. The 87.3% accuracy was measured against the conventional multi-step process, and the report does not print an accuracy figure for that process ^[19]. The tenfold number is also scoped narrowly, to the conversion of light into tokens ^[8]. The model that consumes those tokens, the radio and the airframe all sit outside it.

Miao said there is an opportunity to scale the technology using complementary metal-oxide-semiconductor manufacturing, the process behind smartphone and laptop chips and the sensors in drones ^[14]. The researchers believe that if the scaling works, a drone scanning a disaster zone could fly longer because less energy goes to visual processing ^[15]. Kumar Sokka, CEO of Acre Security, which provides real-world sensing for critical infrastructure and was not involved in the work, described it as a "small-scale demonstration" and said the direction of the research matters to anyone working in the physical world ^[16].