City’s data lost, may need to be entered manually
By Bert Lehman
After the computer system for the city of Clintonville crashed on Oct. 14, the city found out its data was not properly backed up.
The issue was discussed at a special council meeting Oct. 19.
The item was not on the original agenda and was added the day of the meeting. This did not allow for 24-hour notice to the public, as required by state law.
When approving the agenda for the meeting, Mayor Lois Bressette told the council the item was added to the agenda without the 24-hour notice “due to the urgent need to address this issue and the impracticality of convening a separate council meeting and based on this information to ask for a motion to find that good cause exists to consider the issue presented on less than 24-hour notice.” A roll call vote of the council affirmed her statement.
Clintonville City Administrator Chuck Kell informed the council that because the city’s computer systems crashed, a lot of Clintonville Utilities data was lost.
“Our entire utility system is down. It’s time to make utility payments and we have no way of utilizing the system to do that,” Kell said.
This resulted in city staff contacting more than 200 customers via phone to inform them their auto pay system will not release the money to pay their utility bill, Kell said.
“We’ve had a number of system failures over the last several months,” Kell said. “Peggy [Johnson] has lost a lot of emails that have never been able to be recovered. And now this one, this is a very serious situation.”
Kell informed the council that the city sent the computer drives to a company that might be able to restore the data on them. If the company is able to restore the data, it will cost $3,000 to do so.
If the data cannot be restored, city staff will have to manually enter bill and payment information from Aug. 19 to the present for utilities. Accounts payable data was lost since Sept. 28.
Kell said in discussing the issue with city staff and the information technology (IT) consultant the city has on hire, he is receiving conflicting information.
He said the company that supplies the software for the city’s utilities informed him the backups have been failing since Sept. 20. The company wasn’t sure if this was due to disk failures.
Kell said the company informed the city there was something wrong with its computer backup system, but the city’s IT consultant disputed that.
Because of the recent computer problems, Kell told the council he wasn’t comfortable moving forward with only the recommendations of the city’s IT consultant.
He recommended seeking advice from another consultant to do an analysis for the city’s computer system. This would include determining the problems the city currently has, as well as possible solutions. It would cost about $500.
Kell added that he discussed this issue with the city’s current IT consultant and received an email stating backup software was in place that almost immediately backs up information to the Cloud to ensure all data is protected if there is a system failure.
“Well, we’ve had a system failure and we don’t have the data,” Kell said. “Something’s missing here or isn’t working right. And there is nobody else internally who knows what to do with this.”
The request for $4,250, which includes the cost to possibly retrieve lost data, to hire an outside IT consultant to do an analysis of the city’s computer system, and to pay another bill associated with the issue is just the start, Kell said.
“There may be new equipment that we need,” Kell said. “If we have to do this restoration of data by hand internally, there’s going to have to be overtime authorized because the staff isn’t going to have time to do that during the day when their regular job is going on … I don’t know ultimately what the cost of this problem is going to be, but I’d like to at least get going on it to find out what we have to do to solve it.”
Alderman Jim Supanich asked if the city’s IT consultant would have to assume any liability for the lost data.
“Potentially, but I don’t even know if there is a contract,” Kell said. “He’s been around forever, so if there’s anything contractually that addresses that, I’m not aware of it.”
Alderman Brad Rokus told the council that he was uncomfortable discussing some of the issues in open session.
“I’m just a little uneasy having specific conversations about consultant or employee shortcomings in an open session forum,” Rokus said. “I feel it’s not something we would do with a full-time employee if they messed something up that we would be discussing their shortcomings in an open session.”
Alderwoman Amy Steenbock said city data needs to be backed up to alternative sites, and not just to the Cloud.
Clintonville Clerk Treasurer Peggy Johnson stated that she found out information being backed up to the Cloud was only a summary of information, and by error, the city’s database was not being backed up.
Because the backup wasn’t set up properly for the city’s database, Johnson said for the city’s utilities, every change to every account from Aug. 20 to present has to be manually updated. For accounts payable, every invoice the council has approved since Sept. 28 has to be manually recreated.
Prior to the motion, Alderman Lance Bagstad said the city needs to create contracts for these services.
“I don’t know how we’re working with any company without some type of contract in place,” Bagstad said.
He said the contract should be approved at the committee level before moving to the council for approval.
Rokus added that if the contract with the city’s current IT consultant is informal, the city needs to use this opportunity to create a formal contract.
The council ultimately approved authorizing Kell to contract with an outside IT consultant to prepare an analysis and recommendations for changes and upgrades necessary to the city’s computer system and software, to address re-occurring failures up to $750 and costs up to $3,500 for already incurred and future costs to repair or restore data the city has.
Kell provided an update to the Clintonville Tribune-Gazette.
“The data was apparently 100 percent restored but when they returned the data to us and it was reloaded into our system it was not readable by our system. Something to do with the SQL software and us needing to have an SQL expert come and do some programming so it can read the restored data. Our IT service is looking into this and has not given me a status report as of this time as to what we are doing next,” Kell said via email.