Investigators have taken seven samples of food at the school and have sent them for examination, Kumar told CNN, adding he expects a report back Thursday night.
Authorities have gone to the headmistress' house and found stacks of vegetables and grains, Kumar told CNN Thursday. They did not find any pesticides, he said.
"In any Indian home you will find vegetables and grains. These are all allegations," he said. "Everybody is alleging that she is the one responsible and as a professional police officer, I'll collect evidence and only then I will know."
He said the investigation is ongoing and that "if there is a basis for it," authorities will arrest the headmistress.
"But we at least have to speak with her and get her statement," he said.
Kumar told CNN that he was going to the hospital, where he hoped to speak with the ailing children and their families.
Attention is turning to government accountability over the school food program that feeds more than 100 million children, but often with different standards across the country.
Investigators had previously flagged food safety issues at schools in Bihar state, noting in a report that "most schools served often average quality of food items in unhygienic condition."
The food, according to an April report from India's Ministry of Human Resource Development, "was kept in open and dirty ground."
"We had alerted the Bihar government quite a few months back about shortcomings in its food storage. We had expected that the Bihar government would have acted on the recommendations," Human Resources Development Minister Pallam Raju told CNN-IBN. "It is a failure in the system that the rectification has not been done, and I certainly hope that accountability will be called for."
"It is just a reflection of the sort of neglect ... and these sorts of concerns in that state and states around that area," said Reetika Khera of the Indian Institute of Technology in Delhi.
"In the southern part of the country, children get not only good quality food, they also get very nutritious food," she said. "But this is not the case in Bihar."
The school meal program is run by the Indian government in collaboration with state governments, Khera said.
But she said substandard school food is "a reflection of a more general problem, which is the lack of political interest in these programs and -- very importantly -- the lack of accountability."
Education official: We're trying to improve
Shahi, the Bihar state education minister, told CNN that 20 million children receive hot meals in about 73,000 elementary schools.
"We have been endeavoring to improve the quality and ... try to get good food served," he told CNN's Michael Holmes.
"However, the challenge is still there because the magnitude of this program is so huge that there are a number of challenges." He said those challenges are at least partly financial.
"Even though I would unhesitatingly admit that there are some quality issues before us, this is the first incident which has happened in the state," Shahi said. "In the past, we have received complaints regarding quality, but the incident of this nature ... has really shocked us -- shocked the entire state."
What was the poison?
It's unclear whether the children were intentionally or accidentally poisoned.
But officials believe the poison was an organophosphorus compound, a type of chemical that the U.S. Centers for Disease Control and Prevention says is commonly used in agriculture.
It's a nerve agent related to sarin gas, which is used in chemical warfare, the U.S. Health Department says.
Exposure to a high dose can cause an irregular heartbeat, difficulty breathing, paralysis and seizures.